00010000:00010000:3.0:1787826419.674208:0:55606:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab11be lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x32:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82827 expref: -99 pid: 55606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.674239:0:55606:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab11be lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x32:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82827 expref: -99 pid: 55606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.674252:0:55606:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab11be lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x32:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82827 expref: -99 pid: 55606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.674259:0:55606:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab11be lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x32:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82827 expref: -99 pid: 55606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.674267:0:55606:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab11be lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x32:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82827 expref: -99 pid: 55606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826419.674279:0:55606:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff976889eff0c0 x1874670563912832/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 576/720 e 0 to 0 dl 1787826435 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826419.674290:0:55606:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab11be lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x32:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82827 expref: -99 pid: 55606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826419.674301:0:55606:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab11be lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x32:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82827 expref: -99 pid: 55606 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.674313:0:55606:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/50 (0 0) 00000080:00200000:3.0:1787826419.674319:0:55606:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000405:0x32:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826419.674371:0:55606:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x32:0x0]: layout version change: 4294967294 -> 1 00000080:00010000:3.0:1787826419.674377:0:55606:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x32:0x0](ffff976890b3ba88) for lock 0x8e7c99ee62ab11be 00000080:00010000:3.0:1787826419.674381:0:55606:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x32:0x0](ffff976890b3ba88) 00000080:00010000:3.0:1787826419.674384:0:55606:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab11be from it ffffb2d44213bcd8 00010000:00010000:3.0:1787826419.674388:0:55606:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab11be lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x32:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82827 expref: -99 pid: 55606 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.674396:0:55606:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab11be lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x32:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82827 expref: -99 pid: 55606 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826419.674438:0:55606:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x32:0x0] ignore/verify layout 0/1, layout version 21 restore needed 0 00000001:00010000:3.0:1787826419.674447:0:55606:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x32:0x0] 00000008:00010000:3.0:1787826419.674466:0:55606:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1787826419.674469:0:55606:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff9768837ff930, osc lock ffff9768984efe78, flags 0x400001002 00010000:00010000:3.0:1787826419.674515:0:55606:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab11c5 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.674524:0:55606:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab11c5 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787826419.675315:0:55606:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1787826419.680820:0:55606:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x32:0x0] updating i_size 0 i_blocks 0 00000020:00010000:3.0:1787826419.680946:0:55606:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf1638 (ffff976890d10998/0) at cl_lock_release():245 00000020:00010000:3.0:1787826419.680950:0:55606:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf1638 (ffff976890d10998/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826419.680955:0:55606:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768837ff930 (ffff976883760128/1) at cl_lock_cancel():142 00000020:00010000:3.0:1787826419.680959:0:55606:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf1638 (ffff976890d10998/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826419.680961:0:55606:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768837ff930 (ffff976890d10998/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826419.680974:0:55606:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x32:0x0] ignore/verify layout 0/1, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200080:3.0:1787826419.681000:0:55606:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=7ffc5d367684 00000002:00010000:3.0:1787826419.681028:0:55606:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 01004040000001 00010000:00010000:3.0:1787826419.681080:0:55606:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab11cc lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 55606 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.681091:0:55606:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x81000000001000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab11cc lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55606 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787826419.681102:0:55606:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787826419.681121:0:55606:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efd2c0 x1874670563913216/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787826419.681129:0:55606:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab11cc lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x81000000000000 pid: 55606 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826419.681146:0:55606:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55606:1874670563913216:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826419.681329:0:55606:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25900 going to sleep for 11 seconds 00000100:00100000:3.0:1787826419.686564:0:55606:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55606:1874670563913216:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826419.686602:0:55606:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab11cc lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b8283c expref: -99 pid: 55606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.686616:0:55606:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab11cc lrc: 4/1,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b8283c expref: -99 pid: 55606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.686630:0:55606:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab11cc lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b8283c expref: -99 pid: 55606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.686641:0:55606:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab11cc lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b8283c expref: -99 pid: 55606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.686656:0:55606:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab11cc lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b8283c expref: -99 pid: 55606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.686668:0:55606:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab11cc lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b8283c expref: -99 pid: 55606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.686675:0:55606:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab11cc lrc: 4/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b8283c expref: -99 pid: 55606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826419.686687:0:55606:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x600002b, status=0 req@ffff976889efd2c0 x1874670563913216/t8589935032(8589935032) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826435 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826419.686700:0:55606:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efd2c0 x1874670563913216/t8589935032(8589935032) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826435 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826419.686715:0:55606:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab11cc lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b8283c expref: -99 pid: 55606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.686728:0:55606:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 2 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826419.686733:0:55606:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000002:00100000:3.0:1787826419.686755:0:55606:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efd2c0 x1874670563913216/t8589935032(8589935032) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826435 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1787826419.686763:0:55606:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab11cc 00000080:00010000:3.0:1787826419.686769:0:55606:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab11cc from it ffffb2d44213bcc0 00010000:00010000:3.0:1787826419.686772:0:55606:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab11cc lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b8283c expref: -99 pid: 55606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.686780:0:55606:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab11cc lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b8283c expref: -99 pid: 55606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200008:3.0:1787826419.686874:0:55606:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826419.686882:0:55606:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:3.0:1787826419.686896:0:55606:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826419.686907:0:55606:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab11b0 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82812 expref: -99 pid: 55606 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826419.686918:0:55606:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab11b0 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82812 expref: -99 pid: 55606 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826419.686925:0:55606:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826419.686931:0:55606:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:3.0:1787826419.686934:0:55606:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab11b0 00000080:00010000:3.0:1787826419.686939:0:55606:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab11b0 from it ffffb2d44213bcd8 00010000:00010000:3.0:1787826419.686942:0:55606:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab11b0 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82812 expref: -99 pid: 55606 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826419.686950:0:55606:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab11b0 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82812 expref: -99 pid: 55606 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826419.686975:0:55606:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 1 restore needed 0 00000001:00010000:3.0:1787826419.686987:0:55606:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:3.0:1787826419.687007:0:55606:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1787826419.687010:0:55606:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff9768837ff930, osc lock ffff9768984efe78, flags 0x400001002 00010000:00010000:3.0:1787826419.687025:0:55606:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab1132 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82683 expref: -99 pid: 55590 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826419.687043:0:55606:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab1132 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82683 expref: -99 pid: 55590 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826419.687052:0:55606:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab1132 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b82683 expref: -99 pid: 55590 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826419.687061:0:55606:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab1132 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82683 expref: -99 pid: 55590 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826419.687097:0:55606:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab11d3 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.687117:0:55606:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab11d3 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787826419.687786:0:55606:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1787826419.733256:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563913856:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826419.736807:0:2404:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab11e1 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.736884:0:2404:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab11e1 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8285f expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.736902:0:2404:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab11e1 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8285f expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.736915:0:2404:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab11e1 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8285f expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826419.736930:0:2404:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab11e1 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b8285f expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787826419.736942:0:2404:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826419.736958:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab11e1 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b8285f expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.736968:0:2404:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab11e1 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b8285f expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.736976:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab11e1 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b8285f expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.736985:0:2404:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab11e1 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b8285f expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787826419.737005:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563913856:192.168.203.139@tcp:101:lfs.0 00000080:00200008:3.0:1787826419.799506:0:55606:0:(file.c:5224:ll_file_seek()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0], offset: 0, type: (null) 00000080:00200000:3.0:1787826419.799525:0:55606:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1787826419.799536:0:55606:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:3.0:1787826419.799562:0:55606:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1787826419.799567:0:55606:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff9768837fffb0, osc lock ffff9768984ee4e0, flags 0x400001002 00010000:00010000:3.0:1787826419.799593:0:55606:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab11e1 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b8285f expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826419.799626:0:55606:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab11e1 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b8285f expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:3.0:1787826419.799643:0:55606:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1787826419.799647:0:55606:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab11e1 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b8285f expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826419.799656:0:55606:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab11e1 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b8285f expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:3.0:1787826419.799674:0:55606:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1787826419.799685:0:55606:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:3.0:1787826419.799689:0:55606:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9768983a38c8 (ffff976883b9da28/0) at cl_lock_release():245 00000020:00010000:3.0:1787826419.799692:0:55606:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768983a38c8 (ffff976883b9da28/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826419.799695:0:55606:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768837fffb0 (ffff976890d10c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1787826419.799698:0:55606:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768983a38c8 (ffff976883b9da28/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826419.799701:0:55606:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768837fffb0 (ffff976883b9da28/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826419.799714:0:55606:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 21 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1787826419.799721:0:55606:0:(file.c:5262:ll_file_seek()) COMPLETED file f60b.sanity-flr:[0x200000405:0x1d:0x0], offset: 0, type: (null), rc = 0 00000080:00200008:3.0:1787826419.799726:0:55606:0:(file.c:5224:ll_file_seek()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0], offset: 0, type: SEEK_HOLE 00000080:00200000:3.0:1787826419.799730:0:55606:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 21 restore needed 0 00020000:00200000:3.0:1787826419.799736:0:55606:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826419.800074:0:55606:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000080:00200000:3.0:1787826419.800084:0:55606:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826419.800090:0:55606:0:(file.c:1848:ll_io_set_mirror()) f60b.sanity-flr: desiginated mirror: 0 00000080:00200000:3.0:1787826419.800101:0:55606:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826419.800106:0:55606:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826419.800111:0:55606:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 18446744073709551615] 00000008:00010000:3.0:1787826419.800119:0:55606:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 8 00000008:00010000:3.0:1787826419.800121:0:55606:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff9768837fffb0, osc lock ffff9768984ee4e0, flags 0x0 00010000:00010000:3.0:1787826419.800132:0:55606:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab11e1 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b8285f expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826419.800142:0:55606:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab11e1 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b8285f expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826419.800151:0:55606:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab11e1 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b8285f expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826419.800159:0:55606:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab11e1 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b8285f expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826419.800166:0:55606:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab11e1 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b8285f expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826419.800175:0:55606:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:3.0:1787826419.800180:0:55606:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976891f52128 (ffff976883b9da28/0) at cl_lock_release():245 00000020:00010000:3.0:1787826419.800183:0:55606:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976891f52128 (ffff976883b9da28/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826419.800185:0:55606:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768837fffb0 (ffff976890d10c68/1) at cl_lock_cancel():142 00010000:00010000:3.0:1787826419.800189:0:55606:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab11e1 lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b8285f expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826419.800197:0:55606:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab11e1 lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b8285f expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00010000:3.0:1787826419.800205:0:55606:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976891f52128 (ffff976883b9da28/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826419.800207:0:55606:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768837fffb0 (ffff976883b9da28/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826419.800214:0:55606:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 21 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1787826419.800217:0:55606:0:(file.c:5262:ll_file_seek()) COMPLETED file f60b.sanity-flr:[0x200000405:0x1d:0x0], offset: 0, type: SEEK_HOLE, rc = -6 00000080:00200080:3.0:1787826419.800242:0:55606:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=66f4 arg=554 00000080:00200008:3.0:1787826419.800257:0:55606:0:(file.c:2485:do_file_read_iter()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0], ppos: 0, count: 67108864 00000080:00200000:3.0:1787826419.800265:0:55606:0:(file.c:2090:ll_file_io_generic()) f60b.sanity-flr: read ppos: 0, bytes: 67108864 00000080:00200000:3.0:1787826419.800271:0:55606:0:(file.c:1848:ll_io_set_mirror()) f60b.sanity-flr: desiginated mirror: 0 00000020:00200000:3.0:1787826419.800277:0:55606:0:(cl_io.c:218:cl_io_rw_init()) header@00000000a0ca5d5e[0x110, 2, [0x200000405:0x1d:0x0] hash] 00000020:00200000:3.0:1787826419.800279:0:55606:0:(cl_io.c:218:cl_io_rw_init()) io range: 1 [0, 67108864) 2048 0 00000080:00200000:3.0:1787826419.800281:0:55606:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 21 restore needed 0 00000080:00200000:3.0:1787826419.800289:0:55606:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 16383] 00020000:00200000:3.0:1787826419.800295:0:55606:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 67108864) 67108864, 67108864 00020000:00200000:3.0:1787826419.800297:0:55606:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826419.800307:0:55606:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 16383] 00000008:00010000:3.0:1787826419.800313:0:55606:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 11 00000008:00010000:3.0:1787826419.800316:0:55606:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff9768837fffb0, osc lock ffff9768984ee4e0, flags 0x40000 00000080:00200000:3.0:1787826419.800320:0:55606:0:(vvp_io.c:874:vvp_io_read_start()) f60b.sanity-flr: read [0, 67108864) 00000001:00010000:3.0:1787826419.800325:0:55606:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:3.0:1787826419.800330:0:55606:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1787826419.800331:0:55606:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff9768837ffe30, osc lock ffff9768984eeea0, flags 0x400001002 00010000:00010000:3.0:1787826419.800340:0:55606:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab11e1 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b8285f expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826419.800349:0:55606:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab11e1 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b8285f expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00200000:3.0:1787826419.800359:0:55606:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1787826419.800362:0:55606:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab11e1 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b8285f expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826419.800369:0:55606:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab11e1 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b8285f expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00000020:00200000:3.0:1787826419.800378:0:55606:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1787826419.800392:0:55606:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:3.0:1787826419.800394:0:55606:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9768983a38c8 (ffff976883b9da28/0) at cl_lock_release():245 00000020:00010000:3.0:1787826419.800397:0:55606:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768983a38c8 (ffff976883b9da28/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826419.800399:0:55606:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768837ffe30 (ffff976890d10c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1787826419.800402:0:55606:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768983a38c8 (ffff976883b9da28/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826419.800404:0:55606:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768837ffe30 (ffff976883b9da28/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826419.800410:0:55606:0:(vvp_io.c:932:vvp_io_read_start()) count:67108864 iocb pos:0 00000080:00200000:3.0:1787826419.800416:0:55606:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), size=67108864 (max 335544320), offset=0=0x0, pages 16384 (max 81920), lockless, parallel 00000020:00010000:3.0:1787826419.800424:0:55606:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976891f52128 (ffff976883b9da28/0) at cl_lock_release():245 00000020:00010000:3.0:1787826419.800426:0:55606:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976891f52128 (ffff976883b9da28/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826419.800428:0:55606:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768837fffb0 (ffff976890d10c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1787826419.800430:0:55606:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976891f52128 (ffff976883b9da28/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826419.800432:0:55606:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768837fffb0 (ffff976883b9da28/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826419.800436:0:55606:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000020:00200000:3.0:1787826419.800440:0:55606:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1787826419.800443:0:55606:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 16383] 00000080:00200000:3.0:1787826419.800447:0:55606:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 21 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826419.800450:0:55606:0:(file.c:2240:ll_file_io_generic()) f60b.sanity-flr: 1 io complete with rc: 0, result: 0, restart: 0 00000020:00200000:3.0:1787826419.800453:0:55606:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1787826419.800456:0:55606:0:(file.c:2326:ll_file_io_generic()) iot: 1, result: 0 00000080:00200008:3.0:1787826419.800466:0:55606:0:(file.c:5401:ll_fsync()) START file: name=. :VOLATILE:0000:24BD46E6:fd=03, fid=[0x200000405:0x32:0x0], start=0, end=9223372036854775807, datasync=0, need_sync 1/0 00000100:00100000:3.0:1787826419.800519:0:55606:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55606:1874670563913984:192.168.203.139@tcp:44:lfs.0 00000100:00100000:3.0:1787826419.801022:0:55606:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25900 going to sleep for 11 seconds 00000100:00100000:3.0:1787826419.811316:0:55606:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9768877a4800: committing for last_committed 8589935032 gen 1 00000100:00100000:3.0:1787826419.811372:0:55606:0:(client.c:3174:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff976882aec780 x1874670563910912/t8589935027(8589935027) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826435 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1787826419.811713:0:55606:0:(client.c:3174:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff976889ef83c0 x1874670563911296/t8589935028(8589935028) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826435 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1787826419.811723:0:55606:0:(client.c:3174:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff976889efb840 x1874670563911680/t8589935029(8589935029) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826435 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1787826419.811784:0:55606:0:(client.c:3174:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff976889efd2c0 x1874670563913216/t8589935032(8589935032) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826435 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1787826419.811795:0:55606:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55606:1874670563913984:192.168.203.139@tcp:44:lfs.0 00000080:00200000:3.0:1787826419.811817:0:55606:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x32:0x0] ignore/verify layout 0/0, layout version 21 restore needed 0 00020000:00200000:3.0:1787826419.811874:0:55606:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00010000:00010000:1.0:1787826419.812552:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826419.812573:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0162940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670563914240:192.168.203.139@tcp:103:ldlm_bl.0 00000080:00200000:1.0:1787826419.822970:0:55606:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x32:0x0] updating i_size 0 i_blocks 0 00000080:00200000:1.0:1787826419.822985:0:55606:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x32:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200080:1.0:1787826419.835283:0:55606:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=801066da arg=7ffc5d367580 00000080:00200000:1.0:1787826419.835306:0:55606:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:1.0:1787826419.835319:0:55606:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200008:1.0:1787826419.841667:0:55606:0:(llite_lib.c:2427:ll_setattr_raw()) START file . :VOLATILE:0000:24BD46E6:fd=03:[0x200000405:0x32:0x0](ffff976890b3ba88) current size 0, valid attrs 101f0, mode 9768, uid 0, gid 0, new size 4294967296, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826419.000000000 00010000:00010000:1.0:1787826419.841717:0:55606:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab11be lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x32:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b82827 expref: -99 pid: 55606 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787826419.841738:0:55606:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x32:0x0]: losing layout lock 00010000:00010000:1.0:1787826419.841784:0:55606:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab11be lrc: 1/0,0 mode: --/PR res: [0x200000405:0x32:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b82827 expref: -99 pid: 55606 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826419.841792:0:55606:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787826419.841795:0:55606:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab11be lrc: 0/0,0 mode: --/PR res: [0x200000405:0x32:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b82827 expref: -99 pid: 55606 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1787826419.842671:0:55606:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787826419.842685:0:55606:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b0162d00 x1874670563914624/t0(0) o36->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826419.842713:0:55606:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0162d00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55606:1874670563914624:192.168.203.139@tcp:36:lfs.0 00000100:00100000:1.0:1787826419.844440:0:55606:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3b40 going to sleep for 11 seconds 00000100:00100000:1.0:1787826419.861426:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826419.884350:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585262464 00000100:00100000:1.0:1787826419.884360:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585262464 00000100:00100000:1.0:1787826419.884429:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 436 00000100:00100000:1.0:1787826419.884439:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aef840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:10340:x1874670585262464:12345-192.168.203.139@tcp:105: 00010000:00010000:1.0:1787826419.884455:0:53061:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab11ef lrc: 5/0,1 mode: --/PW res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b82882 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826419.884469:0:53061:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab11ef lrc: 5/0,1 mode: --/PW res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b82882 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826419.884479:0:53061:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab11ef lrc: 5/0,1 mode: --/PW res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b82882 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826419.884491:0:53061:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab11ef lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x22153cd396b82882 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826419.884501:0:53061:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab11ef lrc: 6/0,1 mode: PW/PW res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x22153cd396b82882 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826419.884511:0:53061:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab11ef lrc: 6/0,1 mode: PW/PW res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x22153cd396b82882 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826419.885569:0:53061:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff97688925cc00) 00000100:00100000:1.0:1787826419.885638:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aef840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:10340:x1874670585262464:12345-192.168.203.139@tcp:105: Request processed in 1198us (1418us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826419.885648:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 436 00000100:00100000:3.0:1787826419.888675:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563915008:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826419.889372:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563915136:192.168.203.139@tcp:2:lfs.0 00000100:00100000:3.0:1787826419.897722:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563915136:192.168.203.139@tcp:2:lfs.0 00000100:00100000:1.0:1787826419.910297:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:3.0:1787826419.911317:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563915648:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826419.917565:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00010000:00010000:3.0:1787826419.918257:0:55606:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab11f6 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b82890 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.918267:0:55606:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab11f6 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b82890 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.918282:0:55606:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab11f6 lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b82890 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826419.918294:0:55606:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab11f6 lrc: 4/0,2 mode: PW/PW res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x22153cd396b82890 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.918307:0:55606:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab11f6 lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b82890 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.918315:0:55606:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab11f6 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b82890 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826419.918415:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787826419.918431:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563915776:192.168.203.139@tcp:2:lfs.0 00000100:00100000:3.0:1787826419.928659:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563915776:192.168.203.139@tcp:2:lfs.0 00000100:00100000:3.0:1787826419.935380:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563915648:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826419.935736:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826419.935815:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00010000:00010000:1.0:1787826419.939086:0:49796:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1787826419.939096:0:49796:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab1178 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b8276a expref: -99 pid: 55598 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826419.939107:0:49796:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515c00 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:1.0:1787826419.939112:0:49796:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab1178 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b8276a expref: -99 pid: 55598 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826419.939189:0:49796:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab1178 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b8276a expref: -99 pid: 55598 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826419.939199:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab1178 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b8276a expref: -99 pid: 55598 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787826419.939232:0:49796:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x1d:0x0]: losing layout lock 00010000:00010000:1.0:1787826419.939276:0:49796:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab1178 lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b8276a expref: -99 pid: 55598 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826419.939283:0:49796:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787826419.939421:0:49796:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab1178 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b8276a expref: -99 pid: 55598 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826419.939430:0:49796:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab1178 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b8276a expref: -99 pid: 55598 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826419.939444:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787826419.939446:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787826419.939449:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826419.946756:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:3.0:1787826419.948939:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563916672:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826419.953735:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563916672:192.168.203.139@tcp:103:ldlm_bl.0 00000080:00200000:1.0:1787826420.040203:0:55612:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826420.046268:0:55612:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826420.046295:0:55612:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 178 lru_type: 1 00010000:00010000:1.0:1787826420.046312:0:55612:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 178 lru_type: 1 00000080:00200000:1.0:1787826420.046320:0:55612:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826420.046328:0:55612:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826420.046332:0:55612:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826420.046339:0:55612:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44215bbf8 00010000:00010000:1.0:1787826420.046343:0:55612:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 178 lru_type: 1 00010000:00010000:1.0:1787826420.046351:0:55612:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 179 lru_type: 1 00000080:00200000:1.0:1787826420.046366:0:55612:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826420.046376:0:55612:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=27 00000080:00200000:1.0:1787826420.046383:0:55612:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826420.046387:0:55612:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=267 00000080:00200000:1.0:1787826420.046903:0:55612:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826420.046917:0:55612:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826420.046926:0:55612:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 179 lru_type: 1 00010000:00010000:1.0:1787826420.046940:0:55612:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 179 lru_type: 1 00000080:00200000:1.0:1787826420.046947:0:55612:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826420.046952:0:55612:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826420.046955:0:55612:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826420.046960:0:55612:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44215bbf8 00010000:00010000:1.0:1787826420.046963:0:55612:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 179 lru_type: 1 00010000:00010000:1.0:1787826420.046971:0:55612:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 180 lru_type: 1 00000080:00200000:1.0:1787826420.046980:0:55612:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826420.046985:0:55612:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=267 00000080:00200000:1.0:1787826420.046989:0:55612:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 44 00000080:00200008:1.0:1787826420.046996:0:55612:0:(file.c:1090:ll_file_open()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00000080:00200080:1.0:1787826420.047030:0:55612:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8) cmd=c0086915 arg=201f3d0 00000100:00100000:1.0:1787826420.047557:0:55612:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160f00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55612:1874670563917568:192.168.203.139@tcp:34:lfs.0 00000100:00100000:1.0:1787826420.098896:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00010000:00010000:3.0:1787826420.102097:0:49796:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826420.102119:0:49796:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1204 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b828c8 expref: -99 pid: 55608 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826420.102134:0:49796:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514a00 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826420.102141:0:49796:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1204 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b828c8 expref: -99 pid: 55608 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826420.102150:0:49796:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1204 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b828c8 expref: -99 pid: 55608 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826420.102161:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1204 lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b828c8 expref: -99 pid: 55608 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826420.102190:0:49796:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab120b lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b828eb expref: -99 pid: 55614 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826420.102256:0:49796:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1204 lrc: 2/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b828c8 expref: -99 pid: 55608 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826420.102264:0:49796:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826420.102293:0:49796:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1204 lrc: 1/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b828c8 expref: -99 pid: 55608 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826420.102301:0:49796:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1204 lrc: 0/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b828c8 expref: -99 pid: 55608 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826420.102319:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826420.102322:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826420.102324:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826420.102334:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826420.102354:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563918208:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826420.107788:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563918208:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826420.136477:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:3.0:1787826420.138980:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563919232:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826420.143986:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563919232:192.168.203.139@tcp:103:ldlm_bl.0 00000002:00010000:1.0:1787826420.150302:0:55614:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1220 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8292a expref: -99 pid: 55614 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.150316:0:55614:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:1.0:1787826420.150322:0:55614:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1787826420.150340:0:55614:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1220 00000080:00010000:1.0:1787826420.150348:0:55614:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1220 from it ffffb2d44215bbc8 00010000:00010000:1.0:1787826420.150351:0:55614:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1220 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8292a expref: -99 pid: 55614 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.150361:0:55614:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1220 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8292a expref: -99 pid: 55614 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826420.150385:0:55614:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1220 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8292a expref: -99 pid: 55614 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826420.150504:0:55614:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1220 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8292a expref: -99 pid: 55614 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1787826420.150515:0:55614:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1220 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8292a expref: -99 pid: 55614 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787826420.150684:0:55614:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 4294967294 -> 22 00010000:00010000:1.0:1787826420.150690:0:55614:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1220 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b8292a expref: -99 pid: 55614 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826420.150764:0:55614:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1220 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8292a expref: -99 pid: 55614 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787826420.151595:0:55614:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826420.151617:0:55614:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826420.151629:0:55614:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 183 lru_type: 1 00010000:00010000:1.0:1787826420.151644:0:55614:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 183 lru_type: 1 00000080:00200000:1.0:1787826420.151651:0:55614:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826420.151656:0:55614:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826420.151660:0:55614:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826420.151664:0:55614:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44215bbf8 00010000:00010000:1.0:1787826420.151667:0:55614:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 183 lru_type: 1 00010000:00010000:1.0:1787826420.151675:0:55614:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 184 lru_type: 1 00000080:00200000:1.0:1787826420.151685:0:55614:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826420.151692:0:55614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=24 00000080:00200000:1.0:1787826420.151698:0:55614:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826420.151711:0:55614:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787826420.151716:0:55614:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=. :VOLATILE:0000:553BFD62:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), file ffff9768904a1200, open_flags 2f9c2, mode 8180 opened 0 00000080:00200000:1.0:1787826420.151721:0:55614:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=. :VOLATILE:0000:553BFD62:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=open|creat 00000002:00010000:1.0:1787826420.151741:0:55614:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: . :VOLATILE:0000:553BFD62:fd=03,[0x200000405:0x33:0x0]) in obj [0x200000bd0:0x4:0x0], intent: open|creat flags 0574703 00010000:00010000:1.0:1787826420.151792:0:55614:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1227 lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55614 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.151799:0:55614:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1227 lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55614 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1787826420.151812:0:55614:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1787826420.151822:0:55614:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b0161a40 x1874670563919616/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/66640 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1787826420.151865:0:55614:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1227 lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55614 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826420.151883:0:55614:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0161a40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55614:1874670563919616:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826420.152405:0:55614:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3480 going to sleep for 11 seconds 00000100:00100000:1.0:1787826420.166558:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:3.0:1787826420.167818:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef83c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563920256:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826420.172673:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:3.0:1787826420.173883:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef83c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563920256:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826420.240106:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000080:00200000:1.0:1787826420.320273:0:55622:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826420.320303:0:55622:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826420.320322:0:55622:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 189 lru_type: 1 00010000:00010000:1.0:1787826420.320337:0:55622:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 189 lru_type: 1 00000080:00200000:1.0:1787826420.320346:0:55622:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826420.320353:0:55622:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826420.320357:0:55622:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826420.320364:0:55622:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44215bbf8 00010000:00010000:1.0:1787826420.320368:0:55622:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 189 lru_type: 1 00010000:00010000:1.0:1787826420.320384:0:55622:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 190 lru_type: 1 00000080:00200000:1.0:1787826420.320633:0:55622:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826420.320641:0:55622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:1.0:1787826420.320647:0:55622:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826420.320651:0:55622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:1.0:1787826420.320654:0:55622:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 44 00000080:00200008:1.0:1787826420.320661:0:55622:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154400 00000002:00010000:1.0:1787826420.320677:0:55622:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114401 00010000:00010000:1.0:1787826420.320791:0:55622:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab124a lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 55622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.320800:0:55622:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab124a lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55622 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1787826420.320811:0:55622:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1787826420.321315:0:55622:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b01603c0 x1874670563921664/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1787826420.321328:0:55622:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab124a lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55622 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826420.321350:0:55622:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b01603c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55622:1874670563921664:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826420.322154:0:55622:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3c00 going to sleep for 11 seconds 00000100:00100000:1.0:1787826420.328614:0:55622:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b01603c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55622:1874670563921664:192.168.203.139@tcp:101:lfs.0 00010000:00010000:1.0:1787826420.328641:0:55622:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab124a lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b829b6 expref: -99 pid: 55622 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.328656:0:55622:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab124a lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b829b6 expref: -99 pid: 55622 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.328667:0:55622:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab124a lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b829b6 expref: -99 pid: 55622 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787826420.328682:0:55622:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff9768b01603c0 x1874670563921664/t8589935054(8589935054) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826436 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1787826420.328705:0:55622:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9768b01603c0 x1874670563921664/t8589935054(8589935054) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826436 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787826420.328717:0:55622:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab124a lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b829b6 expref: -99 pid: 55622 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787826420.328731:0:55622:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab124a lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b829b6 expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.328745:0:55622:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:1.0:1787826420.328752:0:55622:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1787826420.328803:0:55622:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 23 -> 23 00000080:00010000:1.0:1787826420.328808:0:55622:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab124a 00000080:00010000:1.0:1787826420.328815:0:55622:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab124a from it ffffb2d44215bbf0 00010000:00010000:1.0:1787826420.328819:0:55622:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab124a lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b829b6 expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.328877:0:55622:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab124a lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b829b6 expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000002:00100000:1.0:1787826420.328907:0:55622:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9768b01603c0 x1874670563921664/t8589935054(8589935054) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826436 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1787826420.329050:0:55622:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:1.0:1787826420.329058:0:55622:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:1.0:1787826420.329093:0:55622:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1787826420.329100:0:55622:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:1.0:1787826420.329121:0:55622:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826420.329133:0:55622:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515000/0x8e7c99ee62ab123c lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8298c expref: -99 pid: 55616 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826420.329143:0:55622:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515000/0x8e7c99ee62ab123c lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8298c expref: -99 pid: 55616 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787826420.329149:0:55622:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826420.329155:0:55622:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:1.0:1787826420.329158:0:55622:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab123c 00000080:00010000:1.0:1787826420.329162:0:55622:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab123c from it ffffb2d44215bcd8 00010000:00010000:1.0:1787826420.329165:0:55622:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515000/0x8e7c99ee62ab123c lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b8298c expref: -99 pid: 55616 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826420.329172:0:55622:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515000/0x8e7c99ee62ab123c lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8298c expref: -99 pid: 55616 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787826420.329190:0:55622:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 21 restore needed 0 00000001:00010000:1.0:1787826420.329201:0:55622:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:1.0:1787826420.329227:0:55622:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1787826420.329231:0:55622:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e79030, osc lock ffff976883ff84e0, flags 0x400001002 00010000:00010000:1.0:1787826420.329246:0:55622:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab11ef lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82882 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826420.329256:0:55622:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab11ef lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82882 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826420.329265:0:55622:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab11ef lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b82882 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826420.329273:0:55622:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab11ef lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82882 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826420.329310:0:55622:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cc62ee19/0x8e7c99ee62ab1251 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.329319:0:55622:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cc62ee19/0x8e7c99ee62ab1251 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1787826420.329347:0:55622:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1787826420.333350:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585264384 00000100:00100000:1.0:1787826420.333359:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585264384 00000100:00100000:1.0:1787826420.333455:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 447 00000100:00100000:1.0:1787826420.333465:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aed680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:30695:x1874670585264384:12345-192.168.203.139@tcp:106: 00010000:00010000:1.0:1787826420.333489:0:53061:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab11ef lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82882 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826420.333508:0:53061:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab11ef lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82882 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00010000:1.0:1787826420.333534:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@0000000052c6d642[0x110, 1, [0x200000405:0x32:0x0] hash] 00000020:00010000:1.0:1787826420.333537:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826419 blocks: 0 00000020:00010000:1.0:1787826420.333540:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@00000000bdf6a044[0x116, 2, [0x280000401:0x10a:0x0]] 00000020:00010000:1.0:1787826420.333542:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826419 blocks: 0 00000100:00100000:1.0:1787826420.333596:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aed680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:30695:x1874670585264384:12345-192.168.203.139@tcp:106: Request processed in 133us (335us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826420.333604:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 447 00010000:00010000:1.0:1787826420.339371:0:2402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cc62ee19/0x8e7c99ee62ab1251 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.339394:0:2402:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cc62ee19/0x8e7c99ee62ab1251 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.339778:0:2402:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cc62ee19/0x8e7c99ee62ab1251 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.339787:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cc62ee19/0x8e7c99ee62ab1251 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.339796:0:2402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cc62ee19/0x8e7c99ee62ab1251 lrc: 5/1,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1787826420.339816:0:2402:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:1.0:1787826420.339841:0:2402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1787826420.340085:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cc62ee19/0x8e7c99ee62ab1251 lrc: 4/1,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.340095:0:2402:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cc62ee19/0x8e7c99ee62ab1251 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.340103:0:2402:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cc62ee19/0x8e7c99ee62ab1251 lrc: 4/0,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.340111:0:2402:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976890fdfe00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:1.0:1787826420.340117:0:2402:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cc62ee19/0x8e7c99ee62ab1251 lrc: 5/0,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.340130:0:2402:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1787826420.340137:0:2402:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cc62ee19/0x8e7c99ee62ab1251 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.340146:0:2402:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cc62ee19/0x8e7c99ee62ab1251 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826420.340167:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0163480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563921792:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826420.348859:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585264512 00000100:00100000:1.0:1787826420.348869:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585264512 00000100:00100000:1.0:1787826420.348924:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 448 00000100:00100000:1.0:1787826420.348932:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aecb40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:8247:x1874670585264512:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826420.348950:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515000/0x8e7c99ee62ab123c lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b8298c expref: -99 pid: 55616 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826420.349132:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826420.349140:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aecb40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:8247:x1874670585264512:12345-192.168.203.139@tcp:104: Request processed in 208us (360us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826420.349147:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 448 00000100:00100000:3.0:1787826420.349549:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563922176:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826420.352807:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563922176:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826420.364159:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585264640 00000100:00100000:1.0:1787826420.364168:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585264640 00000100:00100000:1.0:1787826420.364215:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 449 00000100:00100000:1.0:1787826420.364245:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aed2c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:9966:x1874670585264640:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826420.364262:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab125f lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b829e7 expref: -99 pid: 55622 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826420.364420:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826420.364428:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aed2c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:9966:x1874670585264640:12345-192.168.203.139@tcp:104: Request processed in 186us (349us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826420.364447:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 449 00000100:00100000:3.0:1787826420.364744:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563922816:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826420.369330:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563922816:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826420.379800:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:3.0:1787826420.381353:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563923328:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826420.391125:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563923328:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826420.392482:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563923456:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826420.397737:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00010000:00010000:3.0:1787826420.401739:0:2404:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab1274 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.401766:0:2404:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab1274 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.401780:0:2404:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab1274 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.401788:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab1274 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.401795:0:2404:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab1274 lrc: 5/1,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826420.401808:0:2404:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:3.0:1787826420.401814:0:2404:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826420.402032:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab1274 lrc: 4/1,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.402042:0:2404:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab1274 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.402051:0:2404:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab1274 lrc: 4/0,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.402058:0:2404:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515800 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826420.402065:0:2404:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab1274 lrc: 5/0,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.402077:0:2404:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826420.402084:0:2404:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab1274 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.402092:0:2404:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab1274 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826420.402114:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563923456:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826420.410963:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563923712:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826420.413724:0:2403:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab1282 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.413747:0:2403:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab1282 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82a2d expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.413762:0:2403:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab1282 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82a2d expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.413773:0:2403:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab1282 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82a2d expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826420.413788:0:2403:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab1282 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82a2d expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787826420.413800:0:2403:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826420.413819:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab1282 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b82a2d expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.413845:0:2403:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab1282 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b82a2d expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.413854:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab1282 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b82a2d expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.413864:0:2403:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab1282 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b82a2d expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787826420.413891:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563923712:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826420.421006:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563924096:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826420.429022:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00010000:00010000:3.0:1787826420.437139:0:2404:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000023579fe9/0x8e7c99ee62ab1290 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.437162:0:2404:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000023579fe9/0x8e7c99ee62ab1290 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.437176:0:2404:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000023579fe9/0x8e7c99ee62ab1290 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.437189:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000023579fe9/0x8e7c99ee62ab1290 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.437197:0:2404:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000023579fe9/0x8e7c99ee62ab1290 lrc: 5/1,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826420.437211:0:2404:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:3.0:1787826420.437228:0:2404:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826420.437244:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000023579fe9/0x8e7c99ee62ab1290 lrc: 4/1,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.437254:0:2404:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000023579fe9/0x8e7c99ee62ab1290 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.437263:0:2404:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000023579fe9/0x8e7c99ee62ab1290 lrc: 4/0,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.437272:0:2404:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514200 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826420.437278:0:2404:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000023579fe9/0x8e7c99ee62ab1290 lrc: 5/0,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.437289:0:2404:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826420.437296:0:2404:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000023579fe9/0x8e7c99ee62ab1290 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.437308:0:2404:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000023579fe9/0x8e7c99ee62ab1290 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826420.437327:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563924096:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826420.438465:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563924224:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826420.443641:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00010000:00010000:3.0:1787826420.449202:0:2403:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab1297 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.449235:0:2403:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab1297 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.449248:0:2403:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab1297 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.449256:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab1297 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.449264:0:2403:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab1297 lrc: 5/1,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826420.449277:0:2403:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:3.0:1787826420.449284:0:2403:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826420.449303:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab1297 lrc: 4/1,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.449311:0:2403:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab1297 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.449319:0:2403:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab1297 lrc: 4/0,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.449326:0:2403:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515800 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826420.449331:0:2403:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab1297 lrc: 5/0,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.449342:0:2403:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826420.449350:0:2403:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab1297 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.449359:0:2403:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab1297 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826420.449379:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563924224:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826420.449879:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563924352:192.168.203.139@tcp:1:lfs.0 00000100:00100000:1.0:1787826420.453744:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00010000:00010000:3.0:1787826420.454311:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826420.454319:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab11ef lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x22153cd396b82882 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826420.454333:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925cc00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826420.454338:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab11ef lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400020000 nid: local remote: 0x22153cd396b82882 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1787826420.454361:0:42220:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x32:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787826420.454374:0:42220:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x32:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787826420.454425:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab11ef lrc: 2/0,0 mode: --/PW res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x22153cd396b82882 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826420.454432:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826420.454451:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab11ef lrc: 1/0,0 mode: --/PW res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x22153cd396b82882 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826420.454459:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab11ef lrc: 0/0,0 mode: --/PW res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x22153cd396b82882 expref: -99 pid: 55606 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826420.454479:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826420.454481:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826420.454482:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826420.454495:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787826420.454508:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563924480:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826420.458077:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563924480:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826420.458797:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563924352:192.168.203.139@tcp:1:lfs.0 00000100:00100000:3.0:1787826420.459562:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826420.459578:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563924608:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826420.465378:0:2403:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab129e lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.465416:0:2403:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab129e lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82a5e expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.465430:0:2403:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab129e lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82a5e expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.465438:0:2403:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab129e lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82a5e expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826420.465450:0:2403:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab129e lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82a5e expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787826420.465459:0:2403:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826420.465470:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab129e lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b82a5e expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.465476:0:2403:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab129e lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b82a5e expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.465481:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab129e lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b82a5e expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.465487:0:2403:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab129e lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b82a5e expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787826420.465501:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563924608:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826420.524011:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787826420.524033:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563924864:192.168.203.139@tcp:16:lfs.0 00000100:00100000:3.0:1787826420.526035:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563924992:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826420.530483:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563924864:192.168.203.139@tcp:16:lfs.0 00000100:00100000:3.0:1787826420.533796:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563924992:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826420.542865:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563925120:192.168.203.139@tcp:1:lfs.0 00000100:00100000:3.0:1787826420.552238:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563925120:192.168.203.139@tcp:1:lfs.0 00000100:00100000:1.0:1787826420.575546:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585265664 00000100:00100000:1.0:1787826420.575559:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585265664 00000100:00100000:1.0:1787826420.575629:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 456 00000100:00100000:1.0:1787826420.575639:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aecb40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30695:x1874670585265664:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826420.575659:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab1282 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b82a2d expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826420.575746:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826420.575754:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aecb40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30695:x1874670585265664:12345-192.168.203.139@tcp:104: Request processed in 116us (1405us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826420.575762:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 456 00000100:00100000:3.0:1787826420.577025:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826420.578296:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef83c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563925760:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826420.585691:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:3.0:1787826420.589620:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563925888:192.168.203.139@tcp:2:lfs.0 00000100:00100000:3.0:1787826420.599721:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef83c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563925760:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826420.600178:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563925888:192.168.203.139@tcp:2:lfs.0 00000100:00100000:1.0:1787826420.607105:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00010000:00010000:3.0:1787826420.608134:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826420.608142:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab129e lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b82a5e expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826420.608152:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514a00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826420.608190:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab129e lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x22153cd396b82a5e expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1787826420.608230:0:42220:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787826420.608244:0:42220:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787826420.608446:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab129e lrc: 2/0,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b82a5e expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826420.608455:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826420.608663:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab129e lrc: 1/0,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b82a5e expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826420.608673:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab129e lrc: 0/0,0 mode: --/PR res: [0x280000401:0x10a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b82a5e expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826420.608690:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826420.608692:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826420.608694:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826420.614159:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000020:00010000:1.0:1787826420.621749:0:55622:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976891f52128 (ffff976883b9da28/0) at cl_lock_release():245 00000020:00010000:1.0:1787826420.621759:0:55622:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976891f52128 (ffff976883b9da28/0) at cl_lock_cancel():142 00000020:00010000:1.0:1787826420.621765:0:55622:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757cc30 (ffff976890d10c68/1) at cl_lock_cancel():142 00010000:00010000:1.0:1787826420.621776:0:55622:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab12b3 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b82a8f expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.621794:0:55622:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab12b3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82a8f expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:1.0:1787826420.621812:0:55622:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976891f52128 (ffff976883b9da28/0) at cl_lock_fini():72 00000020:00010000:1.0:1787826420.621815:0:55622:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757cc30 (ffff976883b9da28/1) at cl_lock_fini():72 00000080:00200000:1.0:1787826420.622021:0:55622:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 23 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1787826420.622037:0:55622:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8) current size 0, valid attrs 100101f0, mode 9768, uid 0, gid 0, new size 0, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826420.000000000, rc 0 00000080:00200080:1.0:1787826420.622064:0:55622:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=1ae9570 00010000:00010000:1.0:1787826420.622121:0:55622:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab1289 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0x22153cd396b82a3b expref: -99 pid: 55622 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.622139:0:55622:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab1289 lrc: 0/0,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x85809400000000 nid: local remote: 0x22153cd396b82a3b expref: -99 pid: 55622 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787826420.622162:0:55622:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889efad00 x1874670563923968/t8589935059(8589935059) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826436 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1787826420.622174:0:55622:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1787826420.622180:0:55622:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b0161e00 x1874670563926784/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 504/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1787826420.622204:0:55622:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0161e00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55622:1874670563926784:192.168.203.139@tcp:35:lfs.0 00000100:00100000:1.0:1787826420.622546:0:55622:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3e40 going to sleep for 11 seconds 00000100:00100000:1.0:1787826420.625712:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826420.625750:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826420.625779:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585266176 00000100:00100000:1.0:1787826420.625788:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585266176 00000100:00100000:1.0:1787826420.625868:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585266304 00000100:00100000:1.0:1787826420.625871:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585266304 00000100:00100000:1.0:1787826420.625888:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 460 00000100:00100000:1.0:1787826420.625896:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0163840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+6:6522:x1874670585266176:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826420.625914:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab124a lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b829b6 expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1787826420.625978:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826420.625985:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0163840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+6:6522:x1874670585266176:12345-192.168.203.139@tcp:104: Request processed in 90us (283us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826420.625993:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 460 00000100:00100000:1.0:1787826420.626010:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 461 00000100:00100000:1.0:1787826420.626016:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0162940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6522:x1874670585266304:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826420.626024:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab1235 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82969 expref: -99 pid: 55614 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826420.626064:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:1.0:1787826420.626070:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0162940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6522:x1874670585266304:12345-192.168.203.139@tcp:104: Request processed in 54us (319us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826420.626076:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 461 00010000:00010000:3.0:1787826420.626475:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1787826420.626487:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab124a lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b829b6 expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826420.626496:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976890fde800 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826420.626501:0:42221:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab124a lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b829b6 expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826420.626511:0:42221:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab124a lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b829b6 expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826420.626520:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab124a lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 3 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b829b6 expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826420.626545:0:42221:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab1235 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82969 expref: -99 pid: 55614 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000002:00100000:3.0:1787826420.626704:0:42221:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9768b01603c0 x1874670563921664/t8589935054(8589935054) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826436 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826420.626716:0:42221:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 2 is allocated opc 35, max 7 00000100:00100000:3.0:1787826420.626722:0:42221:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efd680 x1874670563927040/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826420.626745:0:42221:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:42221:1874670563927040:192.168.203.139@tcp:35:ldlm_bl.0 00000100:00100000:3.0:1787826420.627114:0:42221:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25900 going to sleep for 11 seconds 00000100:00100000:3.0:1787826420.636265:0:42221:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:42221:1874670563927040:192.168.203.139@tcp:35:ldlm_bl.0 00000002:00100000:3.0:1787826420.636308:0:42221:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9768b01603c0 x1874670563921664/t8589935054(8589935054) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826436 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1787826420.636385:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab124a lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b829b6 expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826420.636394:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826420.636622:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab124a lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b829b6 expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826420.636633:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab124a lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b829b6 expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826420.636655:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826420.636657:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826420.636659:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826420.645177:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00010000:00010000:3.0:1787826420.646130:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826420.646138:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515000/0x8e7c99ee62ab12a5 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x34:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82a7a expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826420.646148:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515000 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826420.646153:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515000/0x8e7c99ee62ab12a5 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x34:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82a7a expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826420.646162:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515000/0x8e7c99ee62ab12a5 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x34:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82a7a expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826420.646171:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515000/0x8e7c99ee62ab12a5 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x34:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b82a7a expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826420.646196:0:42220:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x34:0x0]: losing layout lock 00010000:00010000:1.0:1787826420.647264:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515000/0x8e7c99ee62ab12a5 lrc: 2/0,0 mode: --/CR res: [0x200000405:0x34:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82a7a expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826420.647277:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787826420.647310:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515000/0x8e7c99ee62ab12a5 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x34:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82a7a expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826420.647319:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515000/0x8e7c99ee62ab12a5 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x34:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82a7a expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826420.647341:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787826420.647343:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787826420.647345:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826420.652246:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b01621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670563927424:192.168.203.139@tcp:103:ldlm_bl.0 00000080:00200000:1.0:1787826420.693721:0:55624:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826420.693753:0:55624:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826420.693772:0:55624:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 192 lru_type: 1 00010000:00010000:1.0:1787826420.693787:0:55624:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 192 lru_type: 1 00000080:00200000:1.0:1787826420.693796:0:55624:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826420.693803:0:55624:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826420.693807:0:55624:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826420.693814:0:55624:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44215bb58 00010000:00010000:1.0:1787826420.693818:0:55624:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 192 lru_type: 1 00010000:00010000:1.0:1787826420.693945:0:55624:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 193 lru_type: 1 00000080:00200000:1.0:1787826420.693961:0:55624:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826420.693969:0:55624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=24 00000080:00200000:1.0:1787826420.693975:0:55624:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826420.693985:0:55624:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), flags=8 00000080:00200000:1.0:1787826420.693989:0:55624:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=getattr 00000002:00010000:1.0:1787826420.694002:0:55624:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f60b.sanity-flr,[0x0:0x0:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787826420.694044:0:55624:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab12c1 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55624 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.694119:0:55624:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab12c1 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55624 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.694131:0:55624:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab12c1 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55624 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826420.694158:0:55624:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b01621c0 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55624:1874670563928192:192.168.203.139@tcp:101:getfattr.0 00000100:00100000:1.0:1787826420.694367:0:55624:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3f00 going to sleep for 11 seconds 00000100:00100000:1.0:1787826420.773975:0:55630:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aecb40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55630:1874670563928832:192.168.203.139@tcp:101:lfs.0 00010000:00010000:1.0:1787826420.774000:0:55630:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode CW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab12cf lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82aff expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.774012:0:55630:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab12cf lrc: 4/1,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82aff expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.774023:0:55630:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab12cf lrc: 4/1,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82aff expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.774033:0:55630:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab12cf lrc: 4/1,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82aff expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.774184:0:55630:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab12cf lrc: 5/1,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82aff expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.774194:0:55630:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab12cf lrc: 5/1,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82aff expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.774202:0:55630:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab12cf lrc: 4/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82aff expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787826420.774229:0:55630:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff976882aecb40 x1874670563928832/t8589935069(8589935069) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826436 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1787826420.774242:0:55630:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aecb40 x1874670563928832/t8589935069(8589935069) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826436 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787826420.774258:0:55630:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab12cf lrc: 3/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82aff expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.774272:0:55630:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 8 res 8589935621/29 (0 0) 00000080:00200000:1.0:1787826420.774278:0:55630:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1787826420.774323:0:55630:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x1d:0x0], updating i_size 0 00000080:00010000:1.0:1787826420.774329:0:55630:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab12cf 00000080:00010000:1.0:1787826420.774336:0:55630:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab12cf from it ffffb2d44211bbf0 00010000:00010000:1.0:1787826420.774340:0:55630:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab12cf lrc: 3/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82aff expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.774348:0:55630:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab12cf lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82aff expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:1.0:1787826420.774375:0:55630:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aecb40 x1874670563928832/t8589935069(8589935069) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826436 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:1.0:1787826420.774610:0:55630:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=800466af arg=7ffec64a9c9c 00000080:00200080:1.0:1787826420.774627:0:55630:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=80086601 arg=7ffec64a9c38 00000100:00100000:1.0:1787826420.774669:0:55630:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160f00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55630:1874670563929088:192.168.203.139@tcp:33:lfs.0 00000100:00100000:1.0:1787826420.774790:0:55630:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed30c0 going to sleep for 11 seconds 00000100:00100000:1.0:1787826420.778448:0:55630:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0160f00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55630:1874670563929088:192.168.203.139@tcp:33:lfs.0 00000080:00200080:1.0:1787826420.778656:0:55630:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=7ffec64a9c24 00000002:00010000:1.0:1787826420.778685:0:55630:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 01004040000002 00010000:00010000:1.0:1787826420.778704:0:55630:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab12cf lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x22153cd396b82aff expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826420.778735:0:55630:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925da00/0x8e7c99ee62ab12c8 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b82aea expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787826420.778750:0:55630:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab12ba lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82aa4 expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000002:00100000:1.0:1787826420.778790:0:55630:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976882aed680 x1874670563928576/t8589935068(8589935068) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826436 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1787826420.778802:0:55630:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1787826420.778807:0:55630:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b0160f00 x1874670563929344/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1787826420.778958:0:55630:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160f00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55630:1874670563929344:192.168.203.139@tcp:35:lfs.0 00000100:00100000:1.0:1787826420.785997:0:55630:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0160f00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55630:1874670563929344:192.168.203.139@tcp:35:lfs.0 00000002:00100000:1.0:1787826420.786031:0:55630:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976882aed680 x1874670563928576/t8589935068(8589935068) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826436 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1787826420.786183:0:55630:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab12cf lrc: 1/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b82aff expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826420.786194:0:55630:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925da00/0x8e7c99ee62ab12c8 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b82aea expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787826420.786200:0:55630:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1787826420.786202:0:55630:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab12cf lrc: 0/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b82aff expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826420.786225:0:55630:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925da00/0x8e7c99ee62ab12c8 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b82aea expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787826420.786250:0:55630:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12d6 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 55630 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.786257:0:55630:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x81000000001000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12d6 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 55630 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1787826420.786267:0:55630:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1787826420.786273:0:55630:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b0160f00 x1874670563929600/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 976/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1787826420.786282:0:55630:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12d6 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x81000000000000 pid: 55630 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826420.786301:0:55630:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160f00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55630:1874670563929600:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826420.786592:0:55630:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed30c0 going to sleep for 11 seconds 00000100:00100000:1.0:1787826420.790794:0:55630:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0160f00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55630:1874670563929600:192.168.203.139@tcp:101:lfs.0 00010000:00010000:1.0:1787826420.790856:0:55630:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12d6 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b82b14 expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.790870:0:55630:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12d6 lrc: 4/1,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b82b14 expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.790882:0:55630:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12d6 lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b82b14 expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.790894:0:55630:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12d6 lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b82b14 expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.790907:0:55630:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12d6 lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b82b14 expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.790916:0:55630:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12d6 lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b82b14 expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.790923:0:55630:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12d6 lrc: 4/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b82b14 expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787826420.790936:0:55630:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x600002b, status=0 req@ffff9768b0160f00 x1874670563929600/t8589935071(8589935071) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 976/848 e 0 to 0 dl 1787826436 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1787826420.790948:0:55630:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9768b0160f00 x1874670563929600/t8589935071(8589935071) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 976/848 e 0 to 0 dl 1787826436 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787826420.790961:0:55630:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12d6 lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b82b14 expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.790974:0:55630:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 2 res 8589935621/29 (0 0) 00000080:00200000:1.0:1787826420.790980:0:55630:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000002:00100000:1.0:1787826420.791000:0:55630:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9768b0160f00 x1874670563929600/t8589935071(8589935071) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 768/848 e 0 to 0 dl 1787826436 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1787826420.791009:0:55630:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab12d6 00000080:00010000:1.0:1787826420.791015:0:55630:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab12d6 from it ffffb2d44211bcc0 00010000:00010000:1.0:1787826420.791019:0:55630:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12d6 lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b82b14 expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.791025:0:55630:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12d6 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b82b14 expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200080:1.0:1787826420.791098:0:55630:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=f632e0 00010000:00010000:1.0:1787826420.791141:0:55630:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12d6 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0x22153cd396b82b14 expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.791155:0:55630:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12d6 lrc: 0/0,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x85809400000000 nid: local remote: 0x22153cd396b82b14 expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787826420.791171:0:55630:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9768b0160f00 x1874670563929600/t8589935071(8589935071) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 768/848 e 0 to 0 dl 1787826436 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1787826420.791180:0:55630:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1787826420.791186:0:55630:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b0162580 x1874670563929856/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 504/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1787826420.791208:0:55630:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0162580 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55630:1874670563929856:192.168.203.139@tcp:35:lfs.0 00000100:00100000:1.0:1787826420.791292:0:55630:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed30c0 going to sleep for 11 seconds 00000100:00100000:1.0:1787826420.794551:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585267072 00000100:00100000:1.0:1787826420.794560:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585267072 00000100:00100000:1.0:1787826420.794613:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 464 00000100:00100000:1.0:1787826420.794623:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aee580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585267072:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826420.794640:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab12ba lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82aa4 expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787826420.794730:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826420.794737:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aee580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585267072:12345-192.168.203.139@tcp:104: Request processed in 116us (286us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826420.794745:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 464 00010000:00010000:3.0:1787826420.794805:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826420.794812:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab12ba lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82aa4 expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826420.794847:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514c00 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826420.794853:0:42221:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab12ba lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82aa4 expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826420.794862:0:42221:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab12ba lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82aa4 expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826420.794870:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab12ba lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b82aa4 expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1787826420.794893:0:42221:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x1d:0x0]: losing layout lock 00010000:00010000:3.0:1787826420.794930:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab12ba lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82aa4 expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826420.794937:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826420.794960:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab12ba lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82aa4 expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826420.794968:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab12ba lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82aa4 expref: -99 pid: 55622 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826420.794989:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826420.794991:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826420.794993:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826420.795001:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787826420.795016:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563929984:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826420.798578:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563929984:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826420.801272:0:55630:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0162580 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55630:1874670563929856:192.168.203.139@tcp:35:lfs.0 00000080:00020000:1.0:1787826420.801291:0:55630:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9768877a4800: inode [0x200000405:0x1d:0x0] mdc close failed: rc = -22 00000002:00100000:1.0:1787826420.801309:0:55630:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9768b0160f00 x1874670563929600/t8589935071(8589935071) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 768/848 e 0 to 0 dl 1787826436 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1787826420.801343:0:55630:0:(file.c:7076:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000002:00010000:1.0:1787826420.801354:0:55630:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: layout flags 0 00010000:00010000:1.0:1787826420.801411:0:55630:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12dd lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55630 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.801422:0:55630:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12dd lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55630 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.801431:0:55630:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12dd lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55630 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826420.801452:0:55630:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160f00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55630:1874670563930112:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826420.801529:0:55630:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed30c0 going to sleep for 11 seconds 00000100:00100000:1.0:1787826420.803875:0:55630:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0160f00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55630:1874670563930112:192.168.203.139@tcp:101:lfs.0 00010000:00010000:1.0:1787826420.803908:0:55630:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12dd lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82b29 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.803924:0:55630:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12dd lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82b29 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787826420.803938:0:55630:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9768b0160f00 x1874670563930112/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 376/584 e 0 to 0 dl 1787826436 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787826420.803949:0:55630:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12dd lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82b29 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787826420.803961:0:55630:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12dd lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82b29 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.803973:0:55630:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:1.0:1787826420.803979:0:55630:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1787826420.803996:0:55630:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab12dd 00000080:00010000:1.0:1787826420.804004:0:55630:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab12dd from it ffffb2d44211bbc8 00010000:00010000:1.0:1787826420.804008:0:55630:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12dd lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82b29 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.804018:0:55630:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12dd lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82b29 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826420.804042:0:55630:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12dd lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82b29 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826420.804068:0:55630:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12dd lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82b29 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1787826420.804083:0:55630:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12dd lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82b29 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787826420.804126:0:55630:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 4294967294 -> 24 00010000:00010000:1.0:1787826420.804132:0:55630:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12dd lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82b29 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826420.804139:0:55630:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12dd lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82b29 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000100:00100000:1.0:1787826420.819171:0:53061:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1787826420.819189:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585267200 00000100:00100000:1.0:1787826420.819197:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585267200 00000100:00100000:1.0:1787826420.819292:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 465 00000100:00100000:1.0:1787826420.819302:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768895b9850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6521:x1874670585267200:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826420.819320:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab12dd lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82b29 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826420.819527:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826420.819534:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768895b9850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6521:x1874670585267200:12345-192.168.203.139@tcp:104: Request processed in 233us (474us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826420.819550:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 465 00000100:00100000:3.0:1787826420.821062:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef83c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563931008:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826420.828462:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585267328 00000100:00100000:1.0:1787826420.828471:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585267328 00000100:00100000:1.0:1787826420.828527:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 466 00000100:00100000:1.0:1787826420.828552:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aecf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6521:x1874670585267328:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826420.828570:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab12e4 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82b37 expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826420.828639:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826420.828647:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aecf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6521:x1874670585267328:12345-192.168.203.139@tcp:104: Request processed in 110us (279us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826420.828654:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 466 00010000:00010000:1.0:1787826420.828675:0:49796:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1787826420.828681:0:49796:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab12e4 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82b37 expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826420.828690:0:49796:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925c400 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:1.0:1787826420.828696:0:49796:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab12e4 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x35:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82b37 expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826420.828704:0:49796:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab12e4 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x35:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82b37 expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826420.828712:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab12e4 lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x35:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b82b37 expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787826420.828737:0:49796:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x35:0x0]: losing layout lock 00010000:00010000:1.0:1787826420.828775:0:49796:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab12e4 lrc: 2/0,0 mode: --/PR res: [0x200000405:0x35:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82b37 expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826420.828782:0:49796:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787826420.828799:0:49796:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab12e4 lrc: 1/0,0 mode: --/PR res: [0x200000405:0x35:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82b37 expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826420.828811:0:49796:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab12e4 lrc: 0/0,0 mode: --/PR res: [0x200000405:0x35:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82b37 expref: -99 pid: 55630 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826420.828847:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787826420.828849:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787826420.828850:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826420.828864:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563931136:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826420.834208:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef83c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563931008:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826420.834880:0:55630:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa1c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55630:1874670563930880:192.168.203.139@tcp:35:lfs.0 00000002:00100000:3.0:1787826420.834909:0:55630:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889ef9e00 x1874670563930624/t8589935074(8589935074) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826436 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1787826420.834946:0:55630:0:(file.c:7076:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000002:00010000:3.0:1787826420.834957:0:55630:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: layout flags 0 00010000:00010000:3.0:1787826420.835007:0:55630:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab12f2 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55630 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.835016:0:55630:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab12f2 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55630 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.835026:0:55630:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab12f2 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55630 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826420.835044:0:55630:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efda40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55630:1874670563931264:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826420.835111:0:55630:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25900 going to sleep for 11 seconds 00000100:00100000:1.0:1787826420.835195:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0160780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563931136:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826420.838760:0:55630:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efda40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55630:1874670563931264:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826420.838791:0:55630:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab12f2 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82b68 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.838805:0:55630:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab12f2 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82b68 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826420.838848:0:55630:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff976889efda40 x1874670563931264/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 376/480 e 0 to 0 dl 1787826436 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826420.838861:0:55630:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab12f2 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82b68 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826420.838874:0:55630:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab12f2 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82b68 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.838888:0:55630:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826420.838894:0:55630:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1787826420.838911:0:55630:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab12f2 00000080:00010000:3.0:1787826420.838918:0:55630:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab12f2 from it ffffb2d44211bbc8 00010000:00010000:3.0:1787826420.838923:0:55630:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab12f2 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82b68 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.838932:0:55630:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab12f2 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82b68 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826420.838959:0:55630:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab12f2 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82b68 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826420.838969:0:55630:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab12f2 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82b68 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1787826420.838977:0:55630:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab12f2 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82b68 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826420.839010:0:55630:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab12f2 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82b68 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826420.839017:0:55630:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab12f2 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82b68 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826420.839033:0:55630:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826420.839042:0:55630:0:(lov_io.c:693:lov_io_mirror_init()) [0x200000405:0x1d:0x0]: flr state: 1, move mirror from 1 to 1, have retried: 0, mirror count: 2 00020000:00200000:3.0:1787826420.839045:0:55630:0:(lov_io.c:750:lov_io_mirror_init()) use delayed RPC state for this IO 00020000:00200000:3.0:1787826420.839051:0:55630:0:(lov_io.c:1253:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:3.0:1787826420.839077:0:55630:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:3.0:1787826420.839090:0:55630:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab12f2 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82b68 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826420.839098:0:55630:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab12f2 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82b68 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:3.0:1787826420.839105:0:55630:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab12f2 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82b68 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826420.839120:0:55630:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826420.839124:0:55630:0:(lov_io.c:693:lov_io_mirror_init()) [0x200000405:0x1d:0x0]: flr state: 1, move mirror from 1 to 1, have retried: 0, mirror count: 2 00020000:00200000:3.0:1787826420.839126:0:55630:0:(lov_io.c:750:lov_io_mirror_init()) use delayed RPC state for this IO 00020000:00200000:3.0:1787826420.839128:0:55630:0:(lov_io.c:1253:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:3.0:1787826420.839136:0:55630:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826420.839198:0:55630:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 4294967294 -> 25 00010000:00010000:3.0:1787826420.839203:0:55630:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab12f2 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82b68 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826420.839210:0:55630:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab12f2 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82b68 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:3.0:1787826420.839253:0:55630:0:(file.c:479:ll_file_release()) START file . :VOLATILE:0000:36ADA9F2:fd=03:[0x200000405:0x35:0x0](ffff976890b3ec08), flags 554002 00000002:00100000:3.0:1787826420.839295:0:55630:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976882aec000 x1874670563930368/t8589935073(8589935073) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826436 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826420.839305:0:55630:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1787826420.839310:0:55630:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efda40 x1874670563931520/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826420.839330:0:55630:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efda40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55630:1874670563931520:192.168.203.139@tcp:35:lfs.0 00000100:00100000:3.0:1787826420.839616:0:55630:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25540 going to sleep for 11 seconds 00000100:00100000:3.0:1787826420.842664:0:55630:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efda40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55630:1874670563931520:192.168.203.139@tcp:35:lfs.0 00000002:00100000:3.0:1787826420.842697:0:55630:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976882aec000 x1874670563930368/t8589935073(8589935073) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826436 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1787826420.842753:0:55630:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154002 00000002:00100000:3.0:1787826420.842802:0:55630:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976882aecb40 x1874670563928832/t8589935069(8589935069) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826436 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826420.842811:0:55630:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1787826420.842817:0:55630:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efc780 x1874670563931776/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826420.842935:0:55630:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55630:1874670563931776:192.168.203.139@tcp:35:lfs.0 00000100:00100000:3.0:1787826420.842998:0:55630:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25540 going to sleep for 11 seconds 00000100:00100000:3.0:1787826420.846008:0:55630:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55630:1874670563931776:192.168.203.139@tcp:35:lfs.0 00000080:00200000:1.0:1787826420.859209:0:55632:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826420.859252:0:55632:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826420.859271:0:55632:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 201 lru_type: 1 00010000:00010000:1.0:1787826420.859286:0:55632:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 201 lru_type: 1 00000080:00200000:1.0:1787826420.859293:0:55632:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826420.859300:0:55632:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826420.859303:0:55632:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826420.859310:0:55632:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44213bb58 00010000:00010000:1.0:1787826420.859314:0:55632:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 201 lru_type: 1 00010000:00010000:1.0:1787826420.859322:0:55632:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 202 lru_type: 1 00000080:00200000:1.0:1787826420.859345:0:55632:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826420.859353:0:55632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=24 00000080:00200000:1.0:1787826420.859358:0:55632:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826420.859368:0:55632:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), flags=8 00000080:00200000:1.0:1787826420.859373:0:55632:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=getattr 00000002:00010000:1.0:1787826420.859386:0:55632:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f60b.sanity-flr,[0x0:0x0:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787826420.860698:0:55632:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55632 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.860709:0:55632:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55632 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.860717:0:55632:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55632 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826420.860741:0:55632:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160780 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55632:1874670563931904:192.168.203.139@tcp:101:getfattr.0 00000100:00100000:1.0:1787826420.860844:0:55632:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3f00 going to sleep for 11 seconds 00000100:00100000:1.0:1787826420.866730:0:55632:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0160780 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55632:1874670563931904:192.168.203.139@tcp:101:getfattr.0 00010000:00010000:1.0:1787826420.866757:0:55632:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 4/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82b8b expref: -99 pid: 55632 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.866779:0:55632:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82b8b expref: -99 pid: 55632 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.866787:0:55632:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82b8b expref: -99 pid: 55632 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.866802:0:55632:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82b8b expref: -99 pid: 55632 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.866813:0:55632:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82b8b expref: -99 pid: 55632 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.866847:0:55632:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82b8b expref: -99 pid: 55632 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.866855:0:55632:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82b8b expref: -99 pid: 55632 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.866863:0:55632:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82b8b expref: -99 pid: 55632 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787826420.866875:0:55632:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9768b0160780 x1874670563931904/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 584/720 e 0 to 0 dl 1787826436 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'getfattr.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787826420.866887:0:55632:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82b8b expref: -99 pid: 55632 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787826420.866900:0:55632:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82b8b expref: -99 pid: 55632 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.866910:0:55632:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/29 (0 0) 00000080:00200000:1.0:1787826420.866918:0:55632:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787826420.866963:0:55632:0:(namei.c:147:ll_iget()) got inode: [0x200000405:0x1d:0x0](ffff9768901a80c8): rc = 0 00000080:00200000:1.0:1787826420.866977:0:55632:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 25 -> 25 00000080:00010000:1.0:1787826420.866983:0:55632:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab12f9 00000080:00010000:1.0:1787826420.866993:0:55632:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:1.0:1787826420.866997:0:55632:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab12f9 from it ffffb2d44213bb20 00010000:00010000:1.0:1787826420.867000:0:55632:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82b8b expref: -99 pid: 55632 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.867008:0:55632:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82b8b expref: -99 pid: 55632 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1787826420.867042:0:55632:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1787826420.867048:0:55632:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:1.0:1787826420.867069:0:55632:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826420.867077:0:55632:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82b8b expref: -99 pid: 55632 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826420.867088:0:55632:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82b8b expref: -99 pid: 55632 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787826420.867095:0:55632:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826420.867100:0:55632:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:1.0:1787826420.867103:0:55632:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab12f9 00000080:00010000:1.0:1787826420.867107:0:55632:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab12f9 from it ffffb2d44213bcb8 00010000:00010000:1.0:1787826420.867110:0:55632:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82b8b expref: -99 pid: 55632 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826420.867117:0:55632:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82b8b expref: -99 pid: 55632 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787826420.867142:0:55632:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 23 restore needed 0 00000001:00010000:1.0:1787826420.867152:0:55632:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:1.0:1787826420.867170:0:55632:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1787826420.867173:0:55632:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e791b0, osc lock ffff976883ff9ad0, flags 0x400001002 00010000:00010000:1.0:1787826420.867190:0:55632:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab12ac lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82a81 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826420.867201:0:55632:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab12ac lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82a81 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826420.867211:0:55632:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab12ac lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b82a81 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826420.867232:0:55632:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab12ac lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82a81 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826420.867268:0:55632:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003351033f/0x8e7c99ee62ab1300 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55632 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.867276:0:55632:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003351033f/0x8e7c99ee62ab1300 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55632 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1787826420.867309:0:55632:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1787826420.867328:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670563932032:192.168.203.139@tcp:101:getfattr.0 00000100:00100000:1.0:1787826420.878721:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585267584 00000100:00100000:1.0:1787826420.878730:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585267584 00000100:00100000:1.0:1787826420.878816:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 467 00000100:00100000:1.0:1787826420.878978:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aecb40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:30696:x1874670585267584:12345-192.168.203.139@tcp:106: 00010000:00010000:1.0:1787826420.878995:0:53061:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab12ac lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82a81 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826420.879008:0:53061:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab12ac lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82a81 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:1.0:1787826420.879037:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@00000000004bebc2[0x110, 1, [0x200000405:0x34:0x0] hash] 00000020:00010000:1.0:1787826420.879040:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826420 blocks: 0 00000020:00010000:1.0:1787826420.879044:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@00000000f0cbdb53[0x116, 2, [0x2c0000401:0x10b:0x0]] 00000020:00010000:1.0:1787826420.879045:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826420 blocks: 0 00000100:00100000:1.0:1787826420.879103:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aecb40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:30696:x1874670585267584:12345-192.168.203.139@tcp:106: Request processed in 276us (466us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826420.879110:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 467 00010000:00010000:1.0:1787826420.882695:0:2401:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003351033f/0x8e7c99ee62ab1300 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55632 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.882718:0:2401:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003351033f/0x8e7c99ee62ab1300 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55632 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.882730:0:2401:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003351033f/0x8e7c99ee62ab1300 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55632 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.882739:0:2401:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003351033f/0x8e7c99ee62ab1300 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55632 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.882748:0:2401:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003351033f/0x8e7c99ee62ab1300 lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55632 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1787826420.882761:0:2401:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:1.0:1787826420.882767:0:2401:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1787826420.882786:0:2401:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003351033f/0x8e7c99ee62ab1300 lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55632 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.882794:0:2401:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003351033f/0x8e7c99ee62ab1300 lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55632 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.882803:0:2401:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003351033f/0x8e7c99ee62ab1300 lrc: 4/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55632 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.882810:0:2401:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976890fdec00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:1.0:1787826420.882815:0:2401:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003351033f/0x8e7c99ee62ab1300 lrc: 5/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55632 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.882863:0:2401:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1787826420.882871:0:2401:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003351033f/0x8e7c99ee62ab1300 lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55632 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826420.882880:0:2401:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003351033f/0x8e7c99ee62ab1300 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55632 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826420.882905:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0160780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670563932032:192.168.203.139@tcp:101:getfattr.0 00000080:00200000:1.0:1787826420.913103:0:55636:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826420.913134:0:55636:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826420.913152:0:55636:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 204 lru_type: 1 00010000:00010000:1.0:1787826420.913168:0:55636:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 204 lru_type: 1 00000080:00200000:1.0:1787826420.913176:0:55636:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826420.913183:0:55636:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826420.913186:0:55636:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826420.913193:0:55636:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d442183bf8 00010000:00010000:1.0:1787826420.913197:0:55636:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 204 lru_type: 1 00010000:00010000:1.0:1787826420.913206:0:55636:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 205 lru_type: 1 00000080:00200000:1.0:1787826420.913231:0:55636:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826420.913239:0:55636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=27 00000080:00200000:1.0:1787826420.913245:0:55636:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826420.913249:0:55636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=267 00000080:00200000:1.0:1787826420.913278:0:55636:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826420.913285:0:55636:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826420.913293:0:55636:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 205 lru_type: 1 00010000:00010000:1.0:1787826420.913302:0:55636:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 205 lru_type: 1 00000080:00200000:1.0:1787826420.913309:0:55636:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826420.913313:0:55636:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826420.913316:0:55636:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826420.913320:0:55636:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d442183bf8 00010000:00010000:1.0:1787826420.913322:0:55636:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 205 lru_type: 1 00010000:00010000:1.0:1787826420.913329:0:55636:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 206 lru_type: 1 00000080:00200000:1.0:1787826420.913337:0:55636:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826420.913340:0:55636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=267 00000080:00200000:1.0:1787826420.913344:0:55636:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 44 00000080:00200008:1.0:1787826420.913350:0:55636:0:(file.c:1090:ll_file_open()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00000080:00200080:1.0:1787826420.913380:0:55636:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8) cmd=c0086915 arg=12b53d0 00000100:00100000:1.0:1787826420.913593:0:55636:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55636:1874670563932160:192.168.203.139@tcp:34:lfs.0 00000100:00100000:1.0:1787826420.913739:0:55636:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3e40 going to sleep for 11 seconds 00000100:00100000:1.0:1787826420.919003:0:55636:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0160780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55636:1874670563932160:192.168.203.139@tcp:34:lfs.0 00000080:00200008:1.0:1787826420.919246:0:55636:0:(file.c:479:ll_file_release()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00010000:00010000:1.0:1787826420.919278:0:55636:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9600/0x8e7c99ee62ab0c1c lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b818c9 expref: -99 pid: 55483 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000100:00100000:3.0:1787826420.941693:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563932544:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826420.944177:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00010000:00010000:3.0:1787826420.948175:0:2403:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab130e lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.948203:0:2403:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab130e lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.948228:0:2403:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab130e lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.948236:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab130e lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.948244:0:2403:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab130e lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826420.948258:0:2403:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:3.0:1787826420.948358:0:2403:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826420.948492:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab130e lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.948502:0:2403:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab130e lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.948509:0:2403:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab130e lrc: 4/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.948516:0:2403:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514000 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826420.948522:0:2403:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab130e lrc: 5/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.948533:0:2403:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826420.948551:0:2403:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab130e lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826420.948559:0:2403:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab130e lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826420.948579:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563932544:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826420.954457:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00010000:00010000:3.0:1787826420.954819:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826420.954857:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82b8b expref: -99 pid: 55632 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826420.954866:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976890fdfe00 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826420.954871:0:42221:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82b8b expref: -99 pid: 55632 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826420.954879:0:42221:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82b8b expref: -99 pid: 55632 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826420.954887:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b82b8b expref: -99 pid: 55632 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826420.954913:0:42221:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab1307 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82bb5 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826420.954969:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 2/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82b8b expref: -99 pid: 55632 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826420.954976:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826420.954995:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 1/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82b8b expref: -99 pid: 55632 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826420.955003:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab12f9 lrc: 0/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82b8b expref: -99 pid: 55632 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826420.955017:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826420.955019:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826420.955021:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826420.955028:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826420.955042:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563932928:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826420.961197:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563932928:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826420.971440:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826420.989339:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:3.0:1787826420.992140:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563934080:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826420.999431:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563934080:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826421.000329:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787826421.000345:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563934208:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826421.003108:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00010000:00010000:3.0:1787826421.007946:0:2404:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab1331 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.007969:0:2404:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab1331 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.007983:0:2404:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab1331 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.007991:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab1331 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.007999:0:2404:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab1331 lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826421.008012:0:2404:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:3.0:1787826421.008019:0:2404:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826421.008038:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab1331 lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.008046:0:2404:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab1331 lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.008056:0:2404:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab1331 lrc: 4/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.008064:0:2404:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515a00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826421.008070:0:2404:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab1331 lrc: 5/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.008087:0:2404:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826421.008094:0:2404:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab1331 lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.008103:0:2404:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab1331 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826421.008124:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563934208:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826421.013924:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826421.013943:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563934464:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826421.016882:0:2403:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cf3c1d8e/0x8e7c99ee62ab133f lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.016906:0:2403:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cf3c1d8e/0x8e7c99ee62ab133f lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82c2c expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.016920:0:2403:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cf3c1d8e/0x8e7c99ee62ab133f lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82c2c expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.016932:0:2403:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cf3c1d8e/0x8e7c99ee62ab133f lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82c2c expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826421.016946:0:2403:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cf3c1d8e/0x8e7c99ee62ab133f lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82c2c expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787826421.016963:0:2403:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826421.016980:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cf3c1d8e/0x8e7c99ee62ab133f lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b82c2c expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.016988:0:2403:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cf3c1d8e/0x8e7c99ee62ab133f lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b82c2c expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.016995:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cf3c1d8e/0x8e7c99ee62ab133f lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b82c2c expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.017004:0:2403:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cf3c1d8e/0x8e7c99ee62ab133f lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b82c2c expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787826421.017022:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563934464:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826421.022357:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787826421.022378:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef83c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563934848:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826421.025274:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00010000:00010000:3.0:1787826421.028341:0:2404:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000001aef097c/0x8e7c99ee62ab134d lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.028364:0:2404:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000001aef097c/0x8e7c99ee62ab134d lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.028376:0:2404:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000001aef097c/0x8e7c99ee62ab134d lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.028385:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000001aef097c/0x8e7c99ee62ab134d lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.028602:0:2404:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000001aef097c/0x8e7c99ee62ab134d lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826421.028618:0:2404:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:3.0:1787826421.028625:0:2404:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826421.028686:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000001aef097c/0x8e7c99ee62ab134d lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.028696:0:2404:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000001aef097c/0x8e7c99ee62ab134d lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.028705:0:2404:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000001aef097c/0x8e7c99ee62ab134d lrc: 4/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.028713:0:2404:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515600 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826421.028719:0:2404:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000001aef097c/0x8e7c99ee62ab134d lrc: 5/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.028732:0:2404:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826421.028739:0:2404:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000001aef097c/0x8e7c99ee62ab134d lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.028753:0:2404:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000001aef097c/0x8e7c99ee62ab134d lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826421.028773:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef83c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563934848:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826421.029237:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef83c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563934976:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826421.032359:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00010000:00010000:3.0:1787826421.035334:0:2404:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab1354 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.035354:0:2404:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab1354 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.035367:0:2404:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab1354 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.035376:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab1354 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.035384:0:2404:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab1354 lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826421.035775:0:2404:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:3.0:1787826421.035782:0:2404:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826421.035882:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab1354 lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.035892:0:2404:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab1354 lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.035901:0:2404:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab1354 lrc: 4/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.035909:0:2404:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515a00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826421.035915:0:2404:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab1354 lrc: 5/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.035927:0:2404:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826421.035933:0:2404:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab1354 lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.035942:0:2404:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab1354 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826421.035962:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef83c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563934976:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826421.036202:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563935104:192.168.203.139@tcp:1:lfs.0 00000100:00100000:1.0:1787826421.043027:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00010000:00010000:3.0:1787826421.043364:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826421.043371:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab12ac lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x22153cd396b82a81 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826421.043394:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515e00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826421.043400:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab12ac lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400020000 nid: local remote: 0x22153cd396b82a81 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1787826421.043430:0:42221:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x34:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787826421.043442:0:42221:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x34:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787826421.043489:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab12ac lrc: 2/0,0 mode: --/PW res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x22153cd396b82a81 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826421.043497:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826421.043518:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab12ac lrc: 1/0,0 mode: --/PW res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x22153cd396b82a81 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826421.043527:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab12ac lrc: 0/0,0 mode: --/PW res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x22153cd396b82a81 expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826421.043547:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826421.043549:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826421.043550:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826421.043558:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826421.043569:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563935232:192.168.203.139@tcp:103:ldlm_bl.0 00000080:00200000:1.0:1787826421.048989:0:55638:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:1.0:1787826421.049000:0:55638:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:1.0:1787826421.049027:0:55638:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1787826421.049035:0:55638:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:1.0:1787826421.049054:0:55638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826421.049074:0:55638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab132a lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82c10 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826421.049087:0:55638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab132a lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82c10 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787826421.049094:0:55638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826421.049101:0:55638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:1.0:1787826421.049104:0:55638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab132a 00000080:00010000:1.0:1787826421.049110:0:55638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab132a from it ffffb2d44215bcd8 00010000:00010000:1.0:1787826421.049114:0:55638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab132a lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82c10 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826421.049127:0:55638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab132a lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82c10 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1787826421.049149:0:55638:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 25 restore needed 0 00000001:00010000:1.0:1787826421.049157:0:55638:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:1.0:1787826421.049172:0:55638:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1787826421.049175:0:55638:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e79430, osc lock ffff976883ff8888, flags 0x400001002 00010000:00010000:1.0:1787826421.049232:0:55638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000075a563e1/0x8e7c99ee62ab135b lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.049242:0:55638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000075a563e1/0x8e7c99ee62ab135b lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1787826421.049282:0:55638:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1787826421.049294:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1787826421.049310:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0162940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670563935360:192.168.203.139@tcp:101:lfs.0 00000080:00200000:1.0:1787826421.056387:0:55638:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:1.0:1787826421.056506:0:55638:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf1638 (ffff976890d10998/0) at cl_lock_release():245 00000020:00010000:1.0:1787826421.056510:0:55638:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf1638 (ffff976890d10998/0) at cl_lock_cancel():142 00000020:00010000:1.0:1787826421.056515:0:55638:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e79430 (ffff976883760128/1) at cl_lock_cancel():142 00000020:00010000:1.0:1787826421.056519:0:55638:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf1638 (ffff976890d10998/0) at cl_lock_fini():72 00000020:00010000:1.0:1787826421.056521:0:55638:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e79430 (ffff976890d10998/1) at cl_lock_fini():72 00000080:00200000:1.0:1787826421.056532:0:55638:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 25 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:1.0:1787826421.110641:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00010000:00010000:3.0:1787826421.112807:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826421.112816:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab12b3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000030000 nid: local remote: 0x22153cd396b82a8f expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826421.112851:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514e00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826421.112857:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab12b3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400030000 nid: local remote: 0x22153cd396b82a8f expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826421.112894:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab12b3 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400030020 nid: local remote: 0x22153cd396b82a8f expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826421.112901:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826421.112926:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab12b3 lrc: 1/0,0 mode: --/PW res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400030020 nid: local remote: 0x22153cd396b82a8f expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826421.112935:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab12b3 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400030020 nid: local remote: 0x22153cd396b82a8f expref: -99 pid: 55622 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826421.112957:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826421.112959:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826421.112961:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826421.112969:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826421.112977:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563935744:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826421.117678:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563935744:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826421.117716:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563935616:192.168.203.139@tcp:16:lfs.0 00000100:00100000:1.0:1787826421.131980:0:55638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aecf00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55638:1874670563936128:192.168.203.139@tcp:36:lfs.0 00000080:00200000:1.0:1787826421.132030:0:55638:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x36:0x0] ignore/verify layout 0/0, layout version 25 restore needed 0 00000080:00010000:1.0:1787826421.132053:0:55638:0:(file.c:7076:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000405:0x36:0x0](ffff976890b3e3c8) 00000002:00010000:1.0:1787826421.132065:0:55638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x36:0x0]) in obj [0x200000405:0x36:0x0], intent: layout flags 0 00010000:00010000:1.0:1787826421.132117:0:55638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab1362 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x36:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.132126:0:55638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab1362 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x36:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.132134:0:55638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab1362 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x36:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826421.132153:0:55638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b01621c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55638:1874670563936256:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826421.132239:0:55638:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed30c0 going to sleep for 11 seconds 00000100:00100000:1.0:1787826421.135504:0:55638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b01621c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55638:1874670563936256:192.168.203.139@tcp:101:lfs.0 00010000:00010000:1.0:1787826421.135536:0:55638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab1362 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x36:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82c79 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.135550:0:55638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab1362 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x36:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82c79 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787826421.135564:0:55638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9768b01621c0 x1874670563936256/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 376/480 e 0 to 0 dl 1787826437 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787826421.135575:0:55638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab1362 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x36:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82c79 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787826421.135585:0:55638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab1362 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x36:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82c79 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.135602:0:55638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/54 (0 0) 00000080:00200000:1.0:1787826421.135608:0:55638:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000405:0x36:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1787826421.135624:0:55638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x36:0x0](ffff976890b3e3c8) for lock 0x8e7c99ee62ab1362 00000080:00010000:1.0:1787826421.135631:0:55638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1362 from it ffffb2d44215ba78 00010000:00010000:1.0:1787826421.135635:0:55638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab1362 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x36:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82c79 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.135645:0:55638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab1362 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x36:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82c79 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826421.135669:0:55638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab1362 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x36:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82c79 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826421.135680:0:55638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab1362 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x36:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82c79 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1787826421.135687:0:55638:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x36:0x0](000000008a2623d4) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab1362 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x36:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82c79 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787826421.135722:0:55638:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x36:0x0]: layout version change: 4294967294 -> 1 00010000:00010000:1.0:1787826421.135728:0:55638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab1362 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x36:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82c79 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826421.135735:0:55638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab1362 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x36:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82c79 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:1.0:1787826421.135756:0:55638:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1787826421.135771:0:55638:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:1.0:1787826421.135785:0:55638:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:1.0:1787826421.135789:0:55638:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e79a30, osc lock ffff976883ff8888, flags 0x0 00010000:00010000:1.0:1787826421.135802:0:55638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976889533000 type 11 mode 2 res 10737419265/267 (0 18446744073709551615) 00010000:00010000:1.0:1787826421.135918:0:55638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.135928:0:55638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.135938:0:55638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826421.135962:0:55638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b01621c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55638:1874670563936384:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826421.136070:0:55638:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3e40 going to sleep for 11 seconds 00000100:00100000:1.0:1787826421.139678:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585268992 00000100:00100000:1.0:1787826421.139688:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585268992 00000100:00100000:1.0:1787826421.139746:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 477 00000100:00100000:1.0:1787826421.139755:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aecb40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:14864:x1874670585268992:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826421.139774:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cf3c1d8e/0x8e7c99ee62ab133f lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b82c2c expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826421.140010:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826421.140018:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aecb40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:14864:x1874670585268992:12345-192.168.203.139@tcp:104: Request processed in 264us (695us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826421.140025:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 477 00000100:00100000:3.0:1787826421.140303:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826421.140319:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563936512:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826421.144925:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585269120 00000100:00100000:1.0:1787826421.144935:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585269120 00000100:00100000:1.0:1787826421.144983:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 478 00000100:00100000:1.0:1787826421.144992:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aee1c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:6505:x1874670585269120:12345-192.168.203.139@tcp:105: 00010000:00010000:1.0:1787826421.145008:0:42210:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 4/0,1 mode: --/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.145019:0:42210:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 4/0,1 mode: --/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.145028:0:42210:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 4/0,1 mode: --/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.145040:0:42210:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000000000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.145049:0:42210:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000000000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.145059:0:42210:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.145077:0:42210:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff976890fdf400) 00000100:00100000:1.0:1787826421.145130:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aee1c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:6505:x1874670585269120:12345-192.168.203.139@tcp:105: Request processed in 139us (377us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826421.145137:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 478 00000100:00100000:1.0:1787826421.145560:0:55638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b01621c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55638:1874670563936384:192.168.203.139@tcp:101:lfs.0 00010000:00010000:1.0:1787826421.145580:0:55638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.145597:0:55638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1787826421.145612:0:55638:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.145624:0:55638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.145632:0:55638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826421.148976:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563936512:192.168.203.139@tcp:103:ldlm_bl.0 00000020:00010000:1.0:1787826421.151099:0:55638:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976890fb7998 (ffff976890d10998/0) at cl_lock_release():245 00000020:00010000:1.0:1787826421.151105:0:55638:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976890fb7998 (ffff976890d10998/0) at cl_lock_cancel():142 00000020:00010000:1.0:1787826421.151110:0:55638:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e79a30 (ffff976883760128/1) at cl_lock_cancel():142 00010000:00010000:1.0:1787826421.151120:0:55638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.151136:0:55638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:1.0:1787826421.151152:0:55638:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976890fb7998 (ffff976890d10998/0) at cl_lock_fini():72 00000020:00010000:1.0:1787826421.151155:0:55638:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e79a30 (ffff976890d10998/1) at cl_lock_fini():72 00000080:00200000:1.0:1787826421.151173:0:55638:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x36:0x0] ignore/verify layout 0/0, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1787826421.151187:0:55638:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file . :VOLATILE:0000:54C213A7:fd=03:[0x200000405:0x36:0x0](ffff976890b3e3c8) current size 0, valid attrs 100101f0, mode 9768, uid 0, gid 0, new size 4294967296, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826421.000000000, rc 0 00000080:00200008:1.0:1787826421.151261:0:55638:0:(llite_lib.c:2427:ll_setattr_raw()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8) current size 0, valid attrs 101f0, mode 9768, uid 0, gid 0, new size 0, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826421.000000000 00010000:00010000:1.0:1787826421.151299:0:55638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab132a lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b82c10 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787826421.151321:0:55638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab1307 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82bb5 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1787826421.151369:0:55638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab132a lrc: 1/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b82c10 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787826421.151376:0:55638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787826421.151379:0:55638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab132a lrc: 0/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b82c10 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000020:00100000:1.0:1787826421.151412:0:55638:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787826421.151421:0:55638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b0161e00 x1874670563936896/t0(0) o36->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826421.151444:0:55638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0161e00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55638:1874670563936896:192.168.203.139@tcp:36:lfs.0 00000100:00100000:1.0:1787826421.151528:0:55638:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3e40 going to sleep for 11 seconds 00000100:00100000:1.0:1787826421.157979:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00010000:00010000:3.0:1787826421.158747:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826421.158755:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000075a563e1/0x8e7c99ee62ab135b lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b82c5d expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826421.158766:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976890fde600 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826421.158772:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000075a563e1/0x8e7c99ee62ab135b lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x22153cd396b82c5d expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1787826421.158793:0:42221:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787826421.158806:0:42221:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787826421.158893:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000075a563e1/0x8e7c99ee62ab135b lrc: 2/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b82c5d expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826421.158901:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826421.159163:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000075a563e1/0x8e7c99ee62ab135b lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b82c5d expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826421.159173:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000075a563e1/0x8e7c99ee62ab135b lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b82c5d expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826421.159189:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826421.159191:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826421.159192:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826421.163144:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00010000:00010000:3.0:1787826421.163552:0:55638:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab1370 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b82c8e expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.163563:0:55638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab1370 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b82c8e expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.163578:0:55638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab1370 lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b82c8e expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826421.163591:0:55638:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab1370 lrc: 4/0,2 mode: PW/PW res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x22153cd396b82c8e expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.163604:0:55638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab1370 lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b82c8e expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.163612:0:55638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab1370 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b82c8e expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:1.0:1787826421.174999:0:55638:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976889eda038 (ffff976890d10638/0) at cl_lock_release():245 00000020:00010000:1.0:1787826421.175005:0:55638:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889eda038 (ffff976890d10638/0) at cl_lock_cancel():142 00000020:00010000:1.0:1787826421.175010:0:55638:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e553b0 (ffff97688386c998/1) at cl_lock_cancel():142 00010000:00010000:1.0:1787826421.175022:0:55638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab1370 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b82c8e expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.175038:0:55638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab1370 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82c8e expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:1.0:1787826421.175054:0:55638:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889eda038 (ffff976890d10638/0) at cl_lock_fini():72 00000020:00010000:1.0:1787826421.175057:0:55638:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e553b0 (ffff976890d10638/1) at cl_lock_fini():72 00000080:00200000:1.0:1787826421.175076:0:55638:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 25 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1787826421.175090:0:55638:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8) current size 0, valid attrs 100101f0, mode 9768, uid 0, gid 0, new size 0, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826421.000000000, rc 0 00000080:00200080:1.0:1787826421.175116:0:55638:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=22a6570 00010000:00010000:1.0:1787826421.175168:0:55638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1346 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0x22153cd396b82c3a expref: -99 pid: 55638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.175185:0:55638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1346 lrc: 0/0,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x85809400000000 nid: local remote: 0x22153cd396b82c3a expref: -99 pid: 55638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787826421.175204:0:55638:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889ef9e00 x1874670563934720/t8589935086(8589935086) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826437 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1787826421.175227:0:55638:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1787826421.175233:0:55638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b0160b40 x1874670563937536/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 504/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1787826421.175255:0:55638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160b40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55638:1874670563937536:192.168.203.139@tcp:35:lfs.0 00000100:00100000:1.0:1787826421.176773:0:55638:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3c00 going to sleep for 11 seconds 00000100:00100000:1.0:1787826421.179527:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826421.179606:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826421.179814:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 481 00000100:00100000:1.0:1787826421.179853:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0163c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+6:14257:x1874670585269504:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826421.179872:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab1307 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82bb5 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1787826421.179949:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826421.179956:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0163c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+6:14257:x1874670585269504:12345-192.168.203.139@tcp:104: Request processed in 134us (440us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826421.179963:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 481 00000100:00100000:1.0:1787826421.179980:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 482 00000100:00100000:1.0:1787826421.179987:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0160000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585269632:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826421.179994:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab12f2 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82b68 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826421.180020:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:1.0:1787826421.180025:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0160000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585269632:12345-192.168.203.139@tcp:104: Request processed in 39us (420us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826421.180031:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 482 00010000:00010000:3.0:1787826421.180175:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1787826421.180183:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab1307 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82bb5 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826421.180191:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515c00 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826421.180196:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab1307 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82bb5 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826421.180204:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab1307 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82bb5 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826421.180223:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab1307 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 3 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b82bb5 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826421.180247:0:42220:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab12f2 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82b68 expref: -99 pid: 55630 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000002:00100000:3.0:1787826421.180290:0:42220:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889ef8780 x1874670563932416/t8589935081(8589935081) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826436 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826421.180301:0:42220:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 2 is allocated opc 35, max 7 00000100:00100000:3.0:1787826421.180306:0:42220:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efc780 x1874670563937792/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826421.180328:0:42220:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:42220:1874670563937792:192.168.203.139@tcp:35:ldlm_bl.0 00000100:00100000:3.0:1787826421.180403:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25540 going to sleep for 11 seconds 00000100:00100000:3.0:1787826421.185414:0:42220:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:42220:1874670563937792:192.168.203.139@tcp:35:ldlm_bl.0 00000002:00100000:3.0:1787826421.185445:0:42220:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889ef8780 x1874670563932416/t8589935081(8589935081) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826436 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1787826421.185509:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab1307 lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82bb5 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826421.185519:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826421.185538:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab1307 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82bb5 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826421.185547:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab1307 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82bb5 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826421.185567:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826421.185569:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826421.185571:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826421.190176:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00010000:00010000:3.0:1787826421.190731:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826421.190739:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab1362 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x36:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82c79 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826421.190749:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976890fdec00 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826421.190754:0:42221:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab1362 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x36:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82c79 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826421.190762:0:42221:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab1362 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x36:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82c79 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826421.190770:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab1362 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x36:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b82c79 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826421.190793:0:42221:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x36:0x0]: losing layout lock 00010000:00010000:3.0:1787826421.190864:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab1362 lrc: 2/0,0 mode: --/CR res: [0x200000405:0x36:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82c79 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826421.190871:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826421.190886:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab1362 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x36:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82c79 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826421.190894:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab1362 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x36:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82c79 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826421.190921:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826421.190923:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826421.190925:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00020000:00200000:1.0:1787826421.203974:0:55638:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1787826421.204004:0:55638:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:1.0:1787826421.204067:0:55638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab1377 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82ca3 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826421.204085:0:55638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab1377 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82ca3 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1787826421.204092:0:55638:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab1377 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82ca3 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787826421.204111:0:55638:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 1 restore needed 0 00020000:00200000:1.0:1787826421.204116:0:55638:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1787826421.204125:0:55638:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000008:00010000:1.0:1787826421.204135:0:55638:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 0 [cma]time: 1787826420 1787826412 1787826412 by oinfo size 0 blocks 0 [cma]time 1787826421 1787826412 1787826412 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab1370 lrc: 1/0,0 mode: PW/PW res: [0x2c0000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82c8e expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787826421.204183:0:55638:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 4294967294 -> 26 00010000:00010000:1.0:1787826421.204187:0:55638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab1377 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82ca3 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826421.204192:0:55638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab1377 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82ca3 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:1.0:1787826421.204231:0:55638:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 1154002 00000002:00100000:1.0:1787826421.204276:0:55638:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889ef8f00 x1874670563932800/t8589935082(8589935082) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826436 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1787826421.204284:0:55638:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1787826421.204288:0:55638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b0161a40 x1874670563938560/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1787826421.204309:0:55638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0161a40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55638:1874670563938560:192.168.203.139@tcp:35:lfs.0 00000100:00100000:1.0:1787826421.204380:0:55638:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3300 going to sleep for 11 seconds 00000080:00200000:1.0:1787826421.241668:0:55640:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826421.241701:0:55640:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826421.241719:0:55640:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 209 lru_type: 1 00010000:00010000:1.0:1787826421.241734:0:55640:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 209 lru_type: 1 00000080:00200000:1.0:1787826421.241742:0:55640:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826421.241749:0:55640:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826421.241753:0:55640:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826421.241760:0:55640:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44211bb58 00010000:00010000:1.0:1787826421.241764:0:55640:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 209 lru_type: 1 00010000:00010000:1.0:1787826421.241772:0:55640:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 210 lru_type: 1 00000080:00200000:1.0:1787826421.241788:0:55640:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826421.241795:0:55640:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=24 00000080:00200000:1.0:1787826421.241801:0:55640:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826421.241811:0:55640:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), flags=8 00000080:00200000:1.0:1787826421.241816:0:55640:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=getattr 00000002:00010000:1.0:1787826421.241847:0:55640:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f60b.sanity-flr,[0x0:0x0:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787826421.241898:0:55640:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab137e lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55640 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.241906:0:55640:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab137e lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55640 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.241915:0:55640:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab137e lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55640 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826421.241940:0:55640:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0162d00 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55640:1874670563938944:192.168.203.139@tcp:101:getfattr.0 00000100:00100000:1.0:1787826421.242029:0:55640:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3840 going to sleep for 11 seconds 00000080:00200000:1.0:1787826421.319415:0:55646:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826421.319444:0:55646:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826421.319461:0:55646:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 214 lru_type: 1 00010000:00010000:1.0:1787826421.319478:0:55646:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 214 lru_type: 1 00000080:00200000:1.0:1787826421.319486:0:55646:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826421.319493:0:55646:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826421.319497:0:55646:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826421.319503:0:55646:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44215bbf8 00010000:00010000:1.0:1787826421.319507:0:55646:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 214 lru_type: 1 00010000:00010000:1.0:1787826421.319516:0:55646:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 215 lru_type: 1 00000080:00200000:1.0:1787826421.319539:0:55646:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826421.319547:0:55646:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:1.0:1787826421.319553:0:55646:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826421.319558:0:55646:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:1.0:1787826421.319561:0:55646:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 44 00000080:00200008:1.0:1787826421.319566:0:55646:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154400 00000002:00010000:1.0:1787826421.319581:0:55646:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114401 00010000:00010000:1.0:1787826421.319634:0:55646:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfc00/0x8e7c99ee62ab1385 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 55646 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.319642:0:55646:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfc00/0x8e7c99ee62ab1385 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55646 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1787826421.319654:0:55646:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1787826421.319669:0:55646:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b0162d00 x1874670563939328/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1787826421.319678:0:55646:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfc00/0x8e7c99ee62ab1385 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55646 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826421.319700:0:55646:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0162d00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55646:1874670563939328:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826421.319909:0:55646:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3480 going to sleep for 11 seconds 00000100:00100000:1.0:1787826421.348908:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00010000:00010000:3.0:1787826421.352189:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826421.352203:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab137e lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82cc6 expref: -99 pid: 55640 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826421.352228:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976890fde600 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826421.352234:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab137e lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82cc6 expref: -99 pid: 55640 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826421.352244:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab137e lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82cc6 expref: -99 pid: 55640 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826421.352253:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab137e lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b82cc6 expref: -99 pid: 55640 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826421.352280:0:42220:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfc00/0x8e7c99ee62ab1385 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82ce9 expref: -99 pid: 55646 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826421.352331:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab137e lrc: 2/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82cc6 expref: -99 pid: 55640 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826421.352339:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826421.352372:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab137e lrc: 1/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82cc6 expref: -99 pid: 55640 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826421.352386:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab137e lrc: 0/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82cc6 expref: -99 pid: 55640 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826421.352400:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826421.352401:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826421.352403:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826421.393233:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00010000:00010000:3.0:1787826421.393612:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826421.393619:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab1377 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82ca3 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826421.393628:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925cc00 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826421.393632:0:42221:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab1377 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82ca3 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826421.393639:0:42221:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab1377 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82ca3 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826421.393647:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab1377 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b82ca3 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1787826421.393669:0:42221:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x1d:0x0]: losing layout lock 00010000:00010000:3.0:1787826421.393706:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab1377 lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82ca3 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826421.393712:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826421.393739:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab1377 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82ca3 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826421.393752:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab1377 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82ca3 expref: -99 pid: 55638 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826421.393772:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826421.393773:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826421.393775:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826421.436285:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826421.446286:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826421.452274:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 487 00000100:00100000:1.0:1787826421.452290:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0160f00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6521:x1874670585270656:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826421.452312:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab13a1 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82d36 expref: -99 pid: 55646 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826421.452652:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826421.452660:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0160f00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6521:x1874670585270656:12345-192.168.203.139@tcp:104: Request processed in 374us (6382us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826421.452667:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 487 00010000:00010000:3.0:1787826421.453702:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826421.453714:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab13a1 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82d36 expref: -99 pid: 55646 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826421.453726:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925c600 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826421.453731:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab13a1 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x37:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82d36 expref: -99 pid: 55646 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826421.453740:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab13a1 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x37:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82d36 expref: -99 pid: 55646 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826421.453748:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab13a1 lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x37:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b82d36 expref: -99 pid: 55646 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826421.453772:0:42220:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x37:0x0]: losing layout lock 00010000:00010000:3.0:1787826421.453847:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab13a1 lrc: 2/0,0 mode: --/PR res: [0x200000405:0x37:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82d36 expref: -99 pid: 55646 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826421.453855:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826421.454251:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab13a1 lrc: 1/0,0 mode: --/PR res: [0x200000405:0x37:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82d36 expref: -99 pid: 55646 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826421.454265:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab13a1 lrc: 0/0,0 mode: --/PR res: [0x200000405:0x37:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82d36 expref: -99 pid: 55646 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826421.454287:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826421.454289:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826421.454291:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:1.0:1787826421.534177:0:55648:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826421.534206:0:55648:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826421.534236:0:55648:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 218 lru_type: 1 00010000:00010000:1.0:1787826421.534253:0:55648:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 218 lru_type: 1 00000080:00200000:1.0:1787826421.534262:0:55648:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826421.534270:0:55648:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826421.534273:0:55648:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826421.534280:0:55648:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d442183b58 00010000:00010000:1.0:1787826421.534284:0:55648:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 218 lru_type: 1 00010000:00010000:1.0:1787826421.534293:0:55648:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 219 lru_type: 1 00000080:00200000:1.0:1787826421.534307:0:55648:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826421.534314:0:55648:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=24 00000080:00200000:1.0:1787826421.534320:0:55648:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826421.534329:0:55648:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), flags=8 00000080:00200000:1.0:1787826421.534333:0:55648:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=getattr 00000002:00010000:1.0:1787826421.534346:0:55648:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f60b.sanity-flr,[0x0:0x0:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787826421.534403:0:55648:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55648 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.534412:0:55648:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55648 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.534420:0:55648:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55648 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826421.534445:0:55648:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160780 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55648:1874670563942656:192.168.203.139@tcp:101:getfattr.0 00000100:00100000:1.0:1787826421.535279:0:55648:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3b40 going to sleep for 11 seconds 00000100:00100000:1.0:1787826421.548705:0:55648:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0160780 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55648:1874670563942656:192.168.203.139@tcp:101:getfattr.0 00010000:00010000:1.0:1787826421.548732:0:55648:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 4/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.548759:0:55648:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.548768:0:55648:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.548781:0:55648:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.548792:0:55648:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.548805:0:55648:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.548814:0:55648:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.548821:0:55648:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787826421.548868:0:55648:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9768b0160780 x1874670563942656/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 584/720 e 0 to 0 dl 1787826437 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'getfattr.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787826421.548880:0:55648:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787826421.548893:0:55648:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.548903:0:55648:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/29 (0 0) 00000080:00200000:1.0:1787826421.548909:0:55648:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787826421.548951:0:55648:0:(namei.c:147:ll_iget()) got inode: [0x200000405:0x1d:0x0](ffff9768901a80c8): rc = 0 00000080:00200000:1.0:1787826421.548962:0:55648:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 27 -> 27 00000080:00010000:1.0:1787826421.548967:0:55648:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab13b6 00000080:00010000:1.0:1787826421.548977:0:55648:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:1.0:1787826421.548981:0:55648:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab13b6 from it ffffb2d442183b20 00010000:00010000:1.0:1787826421.548985:0:55648:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.548999:0:55648:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1787826421.549035:0:55648:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1787826421.549041:0:55648:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:1.0:1787826421.549063:0:55648:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826421.549073:0:55648:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826421.549082:0:55648:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787826421.549088:0:55648:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826421.549093:0:55648:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:1.0:1787826421.549095:0:55648:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab13b6 00000080:00010000:1.0:1787826421.549099:0:55648:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab13b6 from it ffffb2d442183cb8 00010000:00010000:1.0:1787826421.549102:0:55648:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826421.549109:0:55648:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787826421.550604:0:55648:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 25 restore needed 0 00000001:00010000:1.0:1787826421.550680:0:55648:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:1.0:1787826421.550702:0:55648:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1787826421.550706:0:55648:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e792b0, osc lock ffff976883ff8888, flags 0x400001002 00010000:00010000:1.0:1787826421.550726:0:55648:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826421.550740:0:55648:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826421.550751:0:55648:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826421.550760:0:55648:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826421.550800:0:55648:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000003888092b/0x8e7c99ee62ab13bd lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55648 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.550809:0:55648:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000003888092b/0x8e7c99ee62ab13bd lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55648 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1787826421.550853:0:55648:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1787826421.558396:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585270912 00000100:00100000:1.0:1787826421.558406:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585270912 00000100:00100000:1.0:1787826421.558461:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 488 00000100:00100000:1.0:1787826421.558470:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aec000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30694:x1874670585270912:12345-192.168.203.139@tcp:106: 00010000:00010000:1.0:1787826421.558495:0:42210:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826421.558509:0:42210:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:1.0:1787826421.558535:0:42210:0:(cl_object.c:322:cl_object_glimpse()) header@00000000be51276b[0x110, 1, [0x200000405:0x36:0x0] hash] 00000020:00010000:1.0:1787826421.558538:0:42210:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826421 blocks: 0 00000020:00010000:1.0:1787826421.558541:0:42210:0:(cl_object.c:322:cl_object_glimpse()) header@000000000e4209da[0x116, 2, [0x280000401:0x10b:0x0]] 00000020:00010000:1.0:1787826421.558543:0:42210:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826421 blocks: 0 00000100:00100000:1.0:1787826421.558596:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aec000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30694:x1874670585270912:12345-192.168.203.139@tcp:106: Request processed in 127us (288us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826421.558604:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 488 00010000:00010000:1.0:1787826421.572874:0:2402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000003888092b/0x8e7c99ee62ab13bd lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55648 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.572898:0:2402:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000003888092b/0x8e7c99ee62ab13bd lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55648 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.572911:0:2402:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000003888092b/0x8e7c99ee62ab13bd lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55648 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.572919:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000003888092b/0x8e7c99ee62ab13bd lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55648 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.572928:0:2402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000003888092b/0x8e7c99ee62ab13bd lrc: 5/1,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55648 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1787826421.572941:0:2402:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:1.0:1787826421.572948:0:2402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1787826421.572966:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000003888092b/0x8e7c99ee62ab13bd lrc: 4/1,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55648 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.572974:0:2402:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000003888092b/0x8e7c99ee62ab13bd lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55648 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.572982:0:2402:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000003888092b/0x8e7c99ee62ab13bd lrc: 4/0,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55648 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.572989:0:2402:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976890fdf200 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:1.0:1787826421.572995:0:2402:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000003888092b/0x8e7c99ee62ab13bd lrc: 5/0,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55648 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.573007:0:2402:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1787826421.573014:0:2402:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000003888092b/0x8e7c99ee62ab13bd lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55648 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.573022:0:2402:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000003888092b/0x8e7c99ee62ab13bd lrc: 0/0,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55648 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826421.573041:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0160780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563942784:192.168.203.139@tcp:101:getfattr.0 00000080:00200000:1.0:1787826421.643685:0:55654:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826421.643719:0:55654:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826421.643738:0:55654:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 223 lru_type: 1 00010000:00010000:1.0:1787826421.643761:0:55654:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 223 lru_type: 1 00000080:00200000:1.0:1787826421.643770:0:55654:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826421.643778:0:55654:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826421.643782:0:55654:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826421.643789:0:55654:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44216bbf8 00010000:00010000:1.0:1787826421.643793:0:55654:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 223 lru_type: 1 00010000:00010000:1.0:1787826421.643801:0:55654:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 224 lru_type: 1 00000080:00200000:1.0:1787826421.644087:0:55654:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826421.644096:0:55654:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:1.0:1787826421.644103:0:55654:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826421.644107:0:55654:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:1.0:1787826421.644111:0:55654:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 44 00000080:00200008:1.0:1787826421.644218:0:55654:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154400 00000002:00010000:1.0:1787826421.644235:0:55654:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114401 00010000:00010000:1.0:1787826421.644287:0:55654:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfa00/0x8e7c99ee62ab13c4 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 55654 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.644296:0:55654:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfa00/0x8e7c99ee62ab13c4 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55654 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1787826421.644307:0:55654:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1787826421.644320:0:55654:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b0160780 x1874670563943168/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1787826421.644329:0:55654:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfa00/0x8e7c99ee62ab13c4 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55654 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826421.644349:0:55654:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55654:1874670563943168:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826421.644721:0:55654:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3180 going to sleep for 11 seconds 00000100:00100000:3.0:1787826421.648333:0:55654:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0160780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55654:1874670563943168:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826421.648366:0:55654:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfa00/0x8e7c99ee62ab13c4 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82db4 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.648381:0:55654:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfa00/0x8e7c99ee62ab13c4 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82db4 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.648449:0:55654:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfa00/0x8e7c99ee62ab13c4 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82db4 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826421.648466:0:55654:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff9768b0160780 x1874670563943168/t8589935108(8589935108) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826437 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826421.648554:0:55654:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9768b0160780 x1874670563943168/t8589935108(8589935108) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826437 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826421.648570:0:55654:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfa00/0x8e7c99ee62ab13c4 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82db4 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826421.648586:0:55654:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfa00/0x8e7c99ee62ab13c4 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82db4 expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.648604:0:55654:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826421.648683:0:55654:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826421.648737:0:55654:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 27 -> 27 00000080:00010000:3.0:1787826421.648742:0:55654:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab13c4 00000080:00010000:3.0:1787826421.648748:0:55654:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab13c4 from it ffffb2d44216bbf0 00010000:00010000:3.0:1787826421.648752:0:55654:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfa00/0x8e7c99ee62ab13c4 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82db4 expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.648762:0:55654:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfa00/0x8e7c99ee62ab13c4 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82db4 expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000002:00100000:3.0:1787826421.648790:0:55654:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9768b0160780 x1874670563943168/t8589935108(8589935108) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826437 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1787826421.649194:0:55654:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:3.0:1787826421.649203:0:55654:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:3.0:1787826421.649244:0:55654:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826421.649252:0:55654:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:3.0:1787826421.649274:0:55654:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826421.649285:0:55654:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826421.649296:0:55654:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826421.649302:0:55654:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826421.649309:0:55654:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:3.0:1787826421.649312:0:55654:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab13b6 00000080:00010000:3.0:1787826421.649316:0:55654:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab13b6 from it ffffb2d44216bcd8 00010000:00010000:3.0:1787826421.649319:0:55654:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826421.649326:0:55654:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826421.649343:0:55654:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1787826421.649353:0:55654:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:3.0:1787826421.649371:0:55654:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1787826421.649375:0:55654:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff9768837ff1b0, osc lock ffff9768984ee4e0, flags 0x400001002 00010000:00010000:3.0:1787826421.650090:0:55654:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826421.650111:0:55654:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826421.650122:0:55654:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826421.650131:0:55654:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826421.650177:0:55654:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000a5185d98/0x8e7c99ee62ab13cb lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.650191:0:55654:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000a5185d98/0x8e7c99ee62ab13cb lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787826421.650371:0:55654:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1787826421.656349:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000080:00200000:3.0:1787826421.666604:0:55654:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:3.0:1787826421.666618:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9768983a3160 (ffff976883b9dea8/0) at cl_lock_release():245 00000020:00010000:3.0:1787826421.666621:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768983a3160 (ffff976883b9dea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826421.666627:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768837ff1b0 (ffff976890d10878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1787826421.666631:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768983a3160 (ffff976883b9dea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826421.666634:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768837ff1b0 (ffff976883b9dea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826421.666648:0:55654:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 27 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1787826421.667014:0:55654:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154000 00010000:00010000:3.0:1787826421.667045:0:55654:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfa00/0x8e7c99ee62ab13c4 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82db4 expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826421.667159:0:55654:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826421.667178:0:55654:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826421.667189:0:55654:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 224 lru_type: 1 00010000:00010000:3.0:1787826421.667199:0:55654:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 224 lru_type: 1 00000080:00200000:3.0:1787826421.667206:0:55654:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826421.667222:0:55654:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826421.667226:0:55654:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826421.667232:0:55654:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44216bbf8 00010000:00010000:3.0:1787826421.667236:0:55654:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 224 lru_type: 1 00010000:00010000:3.0:1787826421.667244:0:55654:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 225 lru_type: 1 00000080:00200000:3.0:1787826421.667259:0:55654:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826421.667265:0:55654:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:3.0:1787826421.667272:0:55654:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787826421.667276:0:55654:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:3.0:1787826421.667281:0:55654:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 46 00000080:00200008:3.0:1787826421.667288:0:55654:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 1154402 00000002:00010000:3.0:1787826421.667302:0:55654:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114403 00010000:00010000:3.0:1787826421.667344:0:55654:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13d2 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 55654 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.667352:0:55654:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13d2 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55654 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787826421.667364:0:55654:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787826421.667376:0:55654:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889eff0c0 x1874670563943552/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787826421.667385:0:55654:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13d2 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55654 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826421.667414:0:55654:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55654:1874670563943552:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826421.669056:0:55654:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25300 going to sleep for 11 seconds 00000100:00100000:1.0:1787826421.673033:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00010000:00010000:3.0:1787826421.676314:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826421.676327:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826421.676340:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976890fdf000 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826421.676347:0:42221:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826421.676356:0:42221:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826421.676365:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826421.676392:0:42221:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfa00/0x8e7c99ee62ab13c4 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82db4 expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826421.676441:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 2/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826421.676448:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826421.676486:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 1/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826421.676495:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf000/0x8e7c99ee62ab13b6 lrc: 0/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82d8a expref: -99 pid: 55648 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826421.676510:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826421.676511:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826421.676514:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826421.690020:0:55654:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55654:1874670563943552:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826421.690050:0:55654:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode CW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13d2 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82dd0 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.690064:0:55654:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13d2 lrc: 4/1,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82dd0 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.690077:0:55654:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13d2 lrc: 4/1,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82dd0 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.690100:0:55654:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13d2 lrc: 4/1,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82dd0 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.690285:0:55654:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13d2 lrc: 5/1,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82dd0 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.690295:0:55654:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13d2 lrc: 5/1,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82dd0 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.690304:0:55654:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13d2 lrc: 4/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82dd0 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826421.690317:0:55654:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff976889eff0c0 x1874670563943552/t8589935109(8589935109) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826437 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826421.690331:0:55654:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889eff0c0 x1874670563943552/t8589935109(8589935109) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826437 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826421.690353:0:55654:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13d2 lrc: 3/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82dd0 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.690367:0:55654:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 8 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826421.690375:0:55654:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000100:00100000:1.0:1787826421.703374:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:3.0:1787826421.714281:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563944320:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826421.720934:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563944320:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826421.729407:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00010000:00010000:3.0:1787826421.730145:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826421.730154:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13d2 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x22153cd396b82dd0 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826421.730163:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976888fb9800 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826421.730168:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13d2 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x22153cd396b82dd0 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826421.730177:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13d2 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x22153cd396b82dd0 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826421.730185:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13d2 lrc: 3/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x22153cd396b82dd0 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826421.730223:0:42220:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfa00/0x8e7c99ee62ab13c4 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82db4 expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826421.730259:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13d2 lrc: 2/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x22153cd396b82dd0 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826421.730271:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826421.730297:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13d2 lrc: 1/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x22153cd396b82dd0 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826421.730305:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13d2 lrc: 0/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x22153cd396b82dd0 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826421.730317:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826421.730319:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826421.730321:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826421.730328:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787826421.730341:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563944832:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826421.739783:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563944832:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826421.749331:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000080:00200000:3.0:1787826421.757051:0:55654:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:3.0:1787826421.757058:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9768983a3160 (ffff976883b9dea8/0) at cl_lock_release():245 00000020:00010000:3.0:1787826421.757061:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768983a3160 (ffff976883b9dea8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826421.757066:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757cfb0 (ffff976890d10638/1) at cl_lock_cancel():142 00000020:00010000:3.0:1787826421.757070:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768983a3160 (ffff976883b9dea8/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826421.757073:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757cfb0 (ffff976883b9dea8/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826421.757086:0:55654:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 27 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1787826421.757109:0:55654:0:(file.c:6313:ll_getattr_dentry()) START file . :VOLATILE:0000:4781A01:fd=03:[0x200000405:0x38:0x0](ffff976890b39148), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826421.757128:0:55654:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x38:0x0](ffff976890b39148),name=. :VOLATILE:0000:4781A01:fd=03 00000002:00010000:3.0:1787826421.757150:0:55654:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x38:0x0]) in obj [0x200000405:0x38:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826421.757201:0:55654:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13f5 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x38:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55654 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.757224:0:55654:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13f5 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x38:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55654 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.757233:0:55654:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13f5 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x38:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55654 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826421.757254:0:55654:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55654:1874670563945088:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826421.757338:0:55654:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25300 going to sleep for 11 seconds 00000100:00100000:3.0:1787826421.771748:0:55654:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55654:1874670563945088:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826421.771774:0:55654:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13f5 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x38:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82e24 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.771788:0:55654:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13f5 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x38:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82e24 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.771800:0:55654:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13f5 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x38:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82e24 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.771811:0:55654:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13f5 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x38:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82e24 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.771916:0:55654:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13f5 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x38:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82e24 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.771926:0:55654:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13f5 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x38:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82e24 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.771940:0:55654:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13f5 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x38:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82e24 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826421.771952:0:55654:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff976889efd2c0 x1874670563945088/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 576/720 e 0 to 0 dl 1787826437 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826421.771965:0:55654:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13f5 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x38:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82e24 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826421.774133:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787826421.774160:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563945216:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826421.789118:0:2404:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab13fc lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.789148:0:2404:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab13fc lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82e2b expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.789165:0:2404:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab13fc lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82e2b expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.789177:0:2404:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab13fc lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82e2b expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826421.789191:0:2404:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab13fc lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82e2b expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787826421.789219:0:2404:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826421.789237:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab13fc lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b82e2b expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.789247:0:2404:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab13fc lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b82e2b expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.789255:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab13fc lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b82e2b expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826421.789264:0:2404:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab13fc lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b82e2b expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787826421.789288:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563945216:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826421.806105:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826421.806140:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563945600:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826421.813333:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585271680 00000100:00100000:1.0:1787826421.813348:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585271680 00000100:00100000:1.0:1787826421.813644:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 494 00000100:00100000:1.0:1787826421.813658:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aec000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:8545:x1874670585271680:12345-192.168.203.139@tcp:106: 00010000:00010000:1.0:1787826421.813675:0:42211:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1787826421.813689:0:42211:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00000020:00010000:1.0:1787826421.813718:0:42211:0:(cl_object.c:322:cl_object_glimpse()) header@00000000be51276b[0x110, 1, [0x200000405:0x36:0x0] hash] 00000020:00010000:1.0:1787826421.813720:0:42211:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826421 blocks: 0 00000020:00010000:1.0:1787826421.813724:0:42211:0:(cl_object.c:322:cl_object_glimpse()) header@000000000e4209da[0x116, 2, [0x280000401:0x10b:0x0]] 00000020:00010000:1.0:1787826421.813726:0:42211:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826421 blocks: 0 00000100:00100000:1.0:1787826421.813794:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aec000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:8545:x1874670585271680:12345-192.168.203.139@tcp:106: Request processed in 141us (3257us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826421.813803:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 494 00000100:00100000:1.0:1787826421.825042:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670563945728:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826421.836918:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585271808 00000100:00100000:1.0:1787826421.836928:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585271808 00000100:00100000:1.0:1787826421.836982:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 495 00000100:00100000:1.0:1787826421.836991:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aed680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:30696:x1874670585271808:12345-192.168.203.139@tcp:106: 00010000:00010000:1.0:1787826421.837007:0:53061:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1787826421.837020:0:53061:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00000020:00010000:1.0:1787826421.837048:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@00000000be51276b[0x110, 1, [0x200000405:0x36:0x0] hash] 00000020:00010000:1.0:1787826421.837050:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826421 blocks: 0 00000020:00010000:1.0:1787826421.837055:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@000000000e4209da[0x116, 2, [0x280000401:0x10b:0x0]] 00000020:00010000:1.0:1787826421.837056:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826421 blocks: 0 00000100:00100000:1.0:1787826421.837111:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aed680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:30696:x1874670585271808:12345-192.168.203.139@tcp:106: Request processed in 121us (585us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826421.837124:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 495 00010000:00010000:1.0:1787826421.847536:0:2401:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab1411 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.847559:0:2401:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab1411 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.847579:0:2401:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab1411 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.847588:0:2401:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab1411 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.847596:0:2401:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab1411 lrc: 5/1,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1787826421.847608:0:2401:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:1.0:1787826421.847615:0:2401:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1787826421.847631:0:2401:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab1411 lrc: 4/1,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.847638:0:2401:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab1411 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.847647:0:2401:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab1411 lrc: 4/0,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.847655:0:2401:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925ce00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:1.0:1787826421.847661:0:2401:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab1411 lrc: 5/0,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.847676:0:2401:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1787826421.847683:0:2401:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab1411 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.847692:0:2401:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab1411 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826421.847716:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670563945728:192.168.203.139@tcp:101:lfs.0 00000080:00200080:0.0:1787826421.848215:0:55654:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=801066da arg=7ffe130aa4c0 00000080:00200000:0.0:1787826421.848226:0:55654:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 27 restore needed 0 00020000:00200000:0.0:1787826421.848237:0:55654:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:1.0:1787826421.848627:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563945856:192.168.203.139@tcp:1:lfs.0 00000100:00100000:0.0:1787826421.855702:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826421.856100:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585271936 00000100:00100000:1.0:1787826421.856123:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585271936 00000100:00100000:1.0:1787826421.856187:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 496 00000100:00100000:1.0:1787826421.856198:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aee940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:14864:x1874670585271936:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826421.856225:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000047ef45fa/0x8e7c99ee62ab1369 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x22153cd396b82c80 expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1787826421.856361:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826421.856368:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aee940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:14864:x1874670585271936:12345-192.168.203.139@tcp:104: Request processed in 173us (676us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826421.856375:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 496 00000100:00100000:3.0:1787826421.859849:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563945984:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826421.871022:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563945856:192.168.203.139@tcp:1:lfs.0 00000080:00200000:1.0:1787826421.871268:0:55654:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:1.0:1787826421.871279:0:55654:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:1.0:1787826421.871309:0:55654:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1787826421.871317:0:55654:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:1.0:1787826421.871337:0:55654:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826421.871356:0:55654:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab13e7 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82e0f expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826421.871369:0:55654:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab13e7 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82e0f expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787826421.871377:0:55654:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826421.871383:0:55654:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:1.0:1787826421.871387:0:55654:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab13e7 00000080:00010000:1.0:1787826421.871394:0:55654:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab13e7 from it ffffb2d44216bcd8 00010000:00010000:1.0:1787826421.871398:0:55654:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab13e7 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82e0f expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826421.871406:0:55654:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab13e7 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82e0f expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1787826421.871428:0:55654:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 27 restore needed 0 00000001:00010000:1.0:1787826421.871447:0:55654:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:1.0:1787826421.871467:0:55654:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1787826421.871470:0:55654:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e79230, osc lock ffff976883ff8ea0, flags 0x400001002 00010000:00010000:1.0:1787826421.871523:0:55654:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826421.871534:0:55654:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826421.871599:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670563946112:192.168.203.139@tcp:101:lfs.0 00000020:00200000:1.0:1787826421.871696:0:55654:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787826421.880160:0:2401:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826421.880188:0:2401:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826421.880204:0:2401:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826421.880225:0:2401:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826421.880239:0:2401:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826421.880251:0:2401:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1787826421.880296:0:2401:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826421.880305:0:2401:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826421.880314:0:2401:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826421.880323:0:2401:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1787826421.880340:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0160f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670563946112:192.168.203.139@tcp:101:lfs.0 00000080:00200000:1.0:1787826421.880583:0:55654:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:1.0:1787826421.880589:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976890e938c8 (ffff97688386c998/0) at cl_lock_release():245 00000020:00010000:1.0:1787826421.880592:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976890e938c8 (ffff97688386c998/0) at cl_lock_cancel():142 00000020:00010000:1.0:1787826421.880597:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e79230 (ffff976890d10908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1787826421.880602:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976890e938c8 (ffff97688386c998/0) at cl_lock_fini():72 00000020:00010000:1.0:1787826421.880605:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e79230 (ffff97688386c998/1) at cl_lock_fini():72 00000080:00200000:1.0:1787826421.880617:0:55654:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 27 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1787826421.926067:0:55654:0:(file.c:5224:ll_file_seek()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0], offset: 0, type: (null) 00000080:00200000:1.0:1787826421.926086:0:55654:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 27 restore needed 0 00000001:00010000:1.0:1787826421.926098:0:55654:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:1.0:1787826421.926119:0:55654:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1787826421.926122:0:55654:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e79230, osc lock ffff976883ff8ea0, flags 0x400001002 00010000:00010000:1.0:1787826421.926148:0:55654:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826421.926165:0:55654:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:1.0:1787826421.926178:0:55654:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1787826421.926183:0:55654:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826421.926193:0:55654:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:1.0:1787826421.926274:0:55654:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1787826421.926284:0:55654:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:1.0:1787826421.926289:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976890e938c8 (ffff97688386c998/0) at cl_lock_release():245 00000020:00010000:1.0:1787826421.926293:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976890e938c8 (ffff97688386c998/0) at cl_lock_cancel():142 00000020:00010000:1.0:1787826421.926296:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e79230 (ffff976890d10908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1787826421.926300:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976890e938c8 (ffff97688386c998/0) at cl_lock_fini():72 00000020:00010000:1.0:1787826421.926303:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e79230 (ffff97688386c998/1) at cl_lock_fini():72 00000080:00200000:1.0:1787826421.926317:0:55654:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 27 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1787826421.926325:0:55654:0:(file.c:5262:ll_file_seek()) COMPLETED file f60b.sanity-flr:[0x200000405:0x1d:0x0], offset: 0, type: (null), rc = 0 00000080:00200008:1.0:1787826421.926330:0:55654:0:(file.c:5224:ll_file_seek()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0], offset: 0, type: SEEK_HOLE 00000080:00200000:1.0:1787826421.926334:0:55654:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 27 restore needed 0 00020000:00200000:1.0:1787826421.926340:0:55654:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1787826421.926578:0:55654:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000080:00200000:1.0:1787826421.926587:0:55654:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1787826421.926593:0:55654:0:(file.c:1848:ll_io_set_mirror()) f60b.sanity-flr: desiginated mirror: 0 00000080:00200000:1.0:1787826421.926595:0:55654:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:1.0:1787826421.926601:0:55654:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1787826421.926605:0:55654:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 18446744073709551615] 00000008:00010000:1.0:1787826421.926612:0:55654:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 8 00000008:00010000:1.0:1787826421.926614:0:55654:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e79230, osc lock ffff976883ff8ea0, flags 0x0 00010000:00010000:1.0:1787826421.926624:0:55654:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826421.926634:0:55654:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826421.926644:0:55654:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826421.926651:0:55654:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826421.926659:0:55654:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787826421.926669:0:55654:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:1.0:1787826421.926674:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885e73998 (ffff97688386c998/0) at cl_lock_release():245 00000020:00010000:1.0:1787826421.926676:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885e73998 (ffff97688386c998/0) at cl_lock_cancel():142 00000020:00010000:1.0:1787826421.926679:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e79230 (ffff976890d10908/1) at cl_lock_cancel():142 00010000:00010000:1.0:1787826421.926682:0:55654:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 4/1,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826421.926690:0:55654:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00010000:1.0:1787826421.926700:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885e73998 (ffff97688386c998/0) at cl_lock_fini():72 00000020:00010000:1.0:1787826421.926702:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e79230 (ffff97688386c998/1) at cl_lock_fini():72 00000080:00200000:1.0:1787826421.926710:0:55654:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 27 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1787826421.926714:0:55654:0:(file.c:5262:ll_file_seek()) COMPLETED file f60b.sanity-flr:[0x200000405:0x1d:0x0], offset: 0, type: SEEK_HOLE, rc = -6 00000080:00200080:1.0:1787826421.926731:0:55654:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=66f4 arg=556 00000080:00200008:1.0:1787826421.926745:0:55654:0:(file.c:2485:do_file_read_iter()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0], ppos: 0, count: 67108864 00000080:00200000:1.0:1787826421.926753:0:55654:0:(file.c:2090:ll_file_io_generic()) f60b.sanity-flr: read ppos: 0, bytes: 67108864 00000080:00200000:1.0:1787826421.926760:0:55654:0:(file.c:1848:ll_io_set_mirror()) f60b.sanity-flr: desiginated mirror: 0 00000020:00200000:1.0:1787826421.926766:0:55654:0:(cl_io.c:218:cl_io_rw_init()) header@00000000a0ca5d5e[0x110, 2, [0x200000405:0x1d:0x0] hash] 00000020:00200000:1.0:1787826421.926768:0:55654:0:(cl_io.c:218:cl_io_rw_init()) io range: 1 [0, 67108864) 2048 0 00000080:00200000:1.0:1787826421.926771:0:55654:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 27 restore needed 0 00000080:00200000:1.0:1787826421.926780:0:55654:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 16383] 00020000:00200000:1.0:1787826421.926786:0:55654:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 67108864) 67108864, 67108864 00020000:00200000:1.0:1787826421.926789:0:55654:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1787826421.926798:0:55654:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 16383] 00000008:00010000:1.0:1787826421.926804:0:55654:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 11 00000008:00010000:1.0:1787826421.926807:0:55654:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e79230, osc lock ffff976883ff8ea0, flags 0x40000 00000080:00200000:1.0:1787826421.926811:0:55654:0:(vvp_io.c:874:vvp_io_read_start()) f60b.sanity-flr: read [0, 67108864) 00000001:00010000:1.0:1787826421.926816:0:55654:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:1.0:1787826421.926819:0:55654:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1787826421.926821:0:55654:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e795b0, osc lock ffff976883ff8750, flags 0x400001002 00010000:00010000:1.0:1787826421.926913:0:55654:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826421.926924:0:55654:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00200000:1.0:1787826421.926932:0:55654:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1787826421.926935:0:55654:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826421.926942:0:55654:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00000020:00200000:1.0:1787826421.926951:0:55654:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1787826421.926955:0:55654:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:1.0:1787826421.926958:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976890e938c8 (ffff97688386c998/0) at cl_lock_release():245 00000020:00010000:1.0:1787826421.926960:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976890e938c8 (ffff97688386c998/0) at cl_lock_cancel():142 00000020:00010000:1.0:1787826421.926962:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e795b0 (ffff976890d10908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1787826421.926965:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976890e938c8 (ffff97688386c998/0) at cl_lock_fini():72 00000020:00010000:1.0:1787826421.926967:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e795b0 (ffff97688386c998/1) at cl_lock_fini():72 00000080:00200000:1.0:1787826421.926973:0:55654:0:(vvp_io.c:932:vvp_io_read_start()) count:67108864 iocb pos:0 00000080:00200000:1.0:1787826421.926979:0:55654:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), size=67108864 (max 335544320), offset=0=0x0, pages 16384 (max 81920), lockless, parallel 00000020:00010000:1.0:1787826421.926987:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885e73998 (ffff97688386c998/0) at cl_lock_release():245 00000020:00010000:1.0:1787826421.926989:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885e73998 (ffff97688386c998/0) at cl_lock_cancel():142 00000020:00010000:1.0:1787826421.926991:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e79230 (ffff976890d10908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1787826421.926994:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885e73998 (ffff97688386c998/0) at cl_lock_fini():72 00000020:00010000:1.0:1787826421.926995:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e79230 (ffff97688386c998/1) at cl_lock_fini():72 00000080:00200000:1.0:1787826421.927000:0:55654:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000020:00200000:1.0:1787826421.927003:0:55654:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1787826421.927006:0:55654:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 16383] 00000080:00200000:1.0:1787826421.927010:0:55654:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 27 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1787826421.927014:0:55654:0:(file.c:2240:ll_file_io_generic()) f60b.sanity-flr: 1 io complete with rc: 0, result: 0, restart: 0 00000020:00200000:1.0:1787826421.927016:0:55654:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1787826421.927019:0:55654:0:(file.c:2326:ll_file_io_generic()) iot: 1, result: 0 00000080:00200008:1.0:1787826421.927029:0:55654:0:(file.c:5401:ll_fsync()) START file: name=. :VOLATILE:0000:4781A01:fd=03, fid=[0x200000405:0x38:0x0], start=0, end=9223372036854775807, datasync=0, need_sync 1/0 00000100:00100000:1.0:1787826421.927085:0:55654:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0163840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55654:1874670563946240:192.168.203.139@tcp:44:lfs.0 00000100:00100000:1.0:1787826421.927242:0:55654:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed36c0 going to sleep for 11 seconds 00000100:00100000:1.0:1787826421.944929:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826421.944969:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585272192 00000100:00100000:1.0:1787826421.944978:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585272192 00000100:00100000:1.0:1787826421.945026:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 497 00000100:00100000:1.0:1787826421.945035:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0161680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30398:x1874670585272192:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826421.945055:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab1370 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000030000 nid: local remote: 0x22153cd396b82c8e expref: -99 pid: 55638 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826421.945120:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826421.945127:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0161680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30398:x1874670585272192:12345-192.168.203.139@tcp:104: Request processed in 93us (208us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826421.945134:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 497 00000100:00100000:0.0:1787826421.946954:0:55654:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9768877a4800: committing for last_committed 8589935113 gen 1 00000100:00100000:0.0:1787826421.946993:0:55654:0:(client.c:3174:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff9768b0160780 x1874670563943168/t8589935108(8589935108) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826437 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1787826421.947003:0:55654:0:(client.c:3174:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff976889eff0c0 x1874670563943552/t8589935109(8589935109) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826437 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1787826421.947010:0:55654:0:(client.c:3174:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff976889efad00 x1874670563943936/t8589935110(8589935110) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826437 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1787826421.947057:0:55654:0:(client.c:3174:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff976889efb0c0 x1874670563945472/t8589935113(8589935113) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826437 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1787826421.947067:0:55654:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0163840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55654:1874670563946240:192.168.203.139@tcp:44:lfs.0 00000080:00200000:0.0:1787826421.947089:0:55654:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x38:0x0] ignore/verify layout 0/0, layout version 23 restore needed 0 00020000:00200000:0.0:1787826421.947101:0:55654:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:0.0:1787826421.947162:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1787826421.947174:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670563946496:192.168.203.139@tcp:16:lfs.0 00000100:00100000:0.0:1787826421.966877:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670563946496:192.168.203.139@tcp:16:lfs.0 00000080:00200000:1.0:1787826421.967237:0:55654:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x38:0x0] updating i_size 0 i_blocks 0 00000080:00200000:1.0:1787826421.967281:0:55654:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x38:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200080:1.0:1787826421.990659:0:55654:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=801066da arg=7ffe130aa4c0 00000080:00200000:1.0:1787826421.990683:0:55654:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 27 restore needed 0 00020000:00200000:1.0:1787826421.990697:0:55654:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:0.0:1787826421.990858:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0162580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670563946624:192.168.203.139@tcp:1:lfs.0 00000100:00100000:0.0:1787826422.000146:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0162580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670563946624:192.168.203.139@tcp:1:lfs.0 00000080:00200008:1.0:1787826422.000278:0:55654:0:(llite_lib.c:2427:ll_setattr_raw()) START file . :VOLATILE:0000:4781A01:fd=03:[0x200000405:0x38:0x0](ffff976890b39148) current size 0, valid attrs 101f0, mode 0, uid 0, gid 0, new size 4294967296, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826421.000000000 00010000:00010000:1.0:1787826422.000317:0:55654:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13f5 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x38:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b82e24 expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787826422.000340:0:55654:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x38:0x0]: losing layout lock 00010000:00010000:1.0:1787826422.000400:0:55654:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13f5 lrc: 1/0,0 mode: --/PR res: [0x200000405:0x38:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b82e24 expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826422.000408:0:55654:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787826422.000412:0:55654:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab13f5 lrc: 0/0,0 mode: --/PR res: [0x200000405:0x38:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b82e24 expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1787826422.000438:0:55654:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787826422.000448:0:55654:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b0162940 x1874670563946880/t0(0) o36->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826422.000474:0:55654:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0162940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55654:1874670563946880:192.168.203.139@tcp:36:lfs.0 00000100:00100000:1.0:1787826422.000573:0:55654:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed36c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826422.018458:0:55654:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0162940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55654:1874670563946880:192.168.203.139@tcp:36:lfs.0 00000080:00200000:0.0:1787826422.018510:0:55654:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x38:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:0.0:1787826422.018535:0:55654:0:(file.c:7076:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000405:0x38:0x0](ffff976890b39148) 00000002:00010000:0.0:1787826422.018546:0:55654:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x38:0x0]) in obj [0x200000405:0x38:0x0], intent: layout flags 0 00010000:00010000:0.0:1787826422.018595:0:55654:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab141f lrc: 3/1,0 mode: --/CR res: [0x200000405:0x38:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55654 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.018606:0:55654:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab141f lrc: 3/1,0 mode: --/CR res: [0x200000405:0x38:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55654 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.018615:0:55654:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab141f lrc: 3/1,0 mode: --/CR res: [0x200000405:0x38:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55654 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826422.018633:0:55654:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed2c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55654:1874670563947008:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826422.018717:0:55654:0:(client.c:2715:ptlrpc_set_wait()) set ffff976891c9a600 going to sleep for 11 seconds 00000100:00100000:0.0:1787826422.022612:0:55654:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed2c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55654:1874670563947008:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826422.022643:0:55654:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab141f lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x38:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82e78 expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.022658:0:55654:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab141f lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x38:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82e78 expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826422.022674:0:55654:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff976882aed2c0 x1874670563947008/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 376/480 e 0 to 0 dl 1787826438 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826422.022693:0:55654:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab141f lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x38:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82e78 expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826422.022706:0:55654:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab141f lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x38:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82e78 expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.022718:0:55654:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/56 (0 0) 00000080:00200000:0.0:1787826422.022725:0:55654:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000405:0x38:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1787826422.022741:0:55654:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x38:0x0](ffff976890b39148) for lock 0x8e7c99ee62ab141f 00000080:00010000:0.0:1787826422.022750:0:55654:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab141f from it ffffb2d44216ba78 00010000:00010000:0.0:1787826422.022754:0:55654:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab141f lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x38:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82e78 expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.022764:0:55654:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab141f lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x38:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82e78 expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826422.022805:0:55654:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab141f lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x38:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82e78 expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826422.022816:0:55654:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab141f lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x38:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82e78 expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826422.022850:0:55654:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x38:0x0](00000000d3f9a416) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab141f lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x38:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82e78 expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826422.022890:0:55654:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x38:0x0]: layout version change: 4294967294 -> 1 00010000:00010000:0.0:1787826422.022897:0:55654:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab141f lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x38:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82e78 expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826422.022906:0:55654:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab141f lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x38:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82e78 expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:0.0:1787826422.022927:0:55654:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826422.022943:0:55654:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:0.0:1787826422.022956:0:55654:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787826422.022961:0:55654:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e55030, osc lock ffff9768849a2c30, flags 0x0 00010000:00010000:0.0:1787826422.022973:0:55654:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9768ad0ae800 type 11 mode 2 res 11811161089/268 (0 18446744073709551615) 00010000:00010000:0.0:1787826422.023007:0:55654:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.023018:0:55654:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.023028:0:55654:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826422.023050:0:55654:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed2c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55654:1874670563947136:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826422.023124:0:55654:0:(client.c:2715:ptlrpc_set_wait()) set ffff976891c9a000 going to sleep for 11 seconds 00000100:00100000:1.0:1787826422.031507:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826422.032544:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585272320 00000100:00100000:0.0:1787826422.032557:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585272320 00000100:00100000:0.0:1787826422.032623:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 498 00000100:00100000:0.0:1787826422.032635:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b01603c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:30695:x1874670585272320:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826422.032653:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab13fc lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b82e2b expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826422.032746:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826422.032753:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b01603c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:30695:x1874670585272320:12345-192.168.203.139@tcp:104: Request processed in 121us (1256us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826422.032760:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 498 00010000:00010000:3.0:1787826422.032873:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826422.032881:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab13fc lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b82e2b expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.032890:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515000 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826422.032897:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab13fc lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x22153cd396b82e2b expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826422.032916:0:42220:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x38:0x0] ignore/verify layout 1/0, layout version 27 restore needed 0 00000080:00200000:3.0:1787826422.032928:0:42220:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x38:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787826422.033031:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab13fc lrc: 2/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b82e2b expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.033040:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826422.033068:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab13fc lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b82e2b expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.033076:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab13fc lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b82e2b expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.033092:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826422.033094:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826422.033096:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826422.033103:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787826422.033117:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563947264:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826422.041626:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826422.042495:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585272448 00000100:00100000:0.0:1787826422.042508:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585272448 00000100:00100000:0.0:1787826422.042557:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 499 00000100:00100000:0.0:1787826422.042567:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0163c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:6503:x1874670585272448:12345-192.168.203.139@tcp:105: 00010000:00010000:0.0:1787826422.042584:0:42211:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 4/0,1 mode: --/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.042597:0:42211:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 4/0,1 mode: --/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.042606:0:42211:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 4/0,1 mode: --/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.042617:0:42211:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000000000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.042626:0:42211:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000000000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.042637:0:42211:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.042656:0:42211:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff97688925ca00) 00000100:00100000:0.0:1787826422.042715:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0163c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:6503:x1874670585272448:12345-192.168.203.139@tcp:105: Request processed in 150us (1097us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826422.042723:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 499 00000100:00100000:0.0:1787826422.045489:0:55654:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed2c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55654:1874670563947136:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826422.045515:0:55654:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.045536:0:55654:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826422.045550:0:55654:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 4/0,2 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.045563:0:55654:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.045571:0:55654:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826422.045662:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1787826422.045678:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670563947392:192.168.203.139@tcp:2:lfs.0 00000100:00100000:3.0:1787826422.052575:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563947264:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826422.060332:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670563947392:192.168.203.139@tcp:2:lfs.0 00000020:00010000:1.0:1787826422.060943:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff97688610f7b8 (ffff976883760c68/0) at cl_lock_release():245 00000020:00010000:1.0:1787826422.060949:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688610f7b8 (ffff976883760c68/0) at cl_lock_cancel():142 00000020:00010000:1.0:1787826422.060956:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e55030 (ffff976883760e18/1) at cl_lock_cancel():142 00010000:00010000:1.0:1787826422.060967:0:55654:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826422.060984:0:55654:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:1.0:1787826422.061001:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688610f7b8 (ffff976883760c68/0) at cl_lock_fini():72 00000020:00010000:1.0:1787826422.061005:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e55030 (ffff976883760c68/1) at cl_lock_fini():72 00000080:00200000:1.0:1787826422.061048:0:55654:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x38:0x0] ignore/verify layout 0/0, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1787826422.061076:0:55654:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file . :VOLATILE:0000:4781A01:fd=03:[0x200000405:0x38:0x0](ffff976890b39148) current size 0, valid attrs 100101f0, mode 0, uid 0, gid 0, new size 4294967296, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826421.000000000, rc 0 00000080:00200008:1.0:1787826422.061134:0:55654:0:(llite_lib.c:2427:ll_setattr_raw()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8) current size 0, valid attrs 101f0, mode 9768, uid 0, gid 0, new size 0, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826422.000000000 00010000:00010000:1.0:1787826422.061167:0:55654:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab13e7 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b82e0f expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787826422.061189:0:55654:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfa00/0x8e7c99ee62ab13c4 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82db4 expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1787826422.061246:0:55654:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab13e7 lrc: 1/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b82e0f expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787826422.061254:0:55654:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787826422.061256:0:55654:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab13e7 lrc: 0/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b82e0f expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000020:00100000:1.0:1787826422.061281:0:55654:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787826422.061292:0:55654:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b0160000 x1874670563947648/t0(0) o36->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826422.061314:0:55654:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160000 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55654:1874670563947648:192.168.203.139@tcp:36:lfs.0 00000100:00100000:1.0:1787826422.061421:0:55654:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3240 going to sleep for 11 seconds 00000100:00100000:0.0:1787826422.076217:0:55654:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0160000 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55654:1874670563947648:192.168.203.139@tcp:36:lfs.0 00000080:00200000:0.0:1787826422.076264:0:55654:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1787826422.076280:0:55654:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826422.076296:0:55654:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:0.0:1787826422.076313:0:55654:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787826422.076318:0:55654:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e554b0, osc lock ffff9768849a3860, flags 0x0 00010000:00010000:0.0:1787826422.076336:0:55654:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976889533000 type 11 mode 2 res 10737419265/267 (0 18446744073709551615) 00010000:00010000:0.0:1787826422.076397:0:55654:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab142d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.076410:0:55654:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab142d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.076421:0:55654:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab142d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826422.076444:0:55654:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee1c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55654:1874670563947776:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826422.076555:0:55654:0:(client.c:2715:ptlrpc_set_wait()) set ffff976891c9a000 going to sleep for 11 seconds 00000100:00100000:1.0:1787826422.088567:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826422.088657:0:42210:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:0.0:1787826422.088676:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585272576 00000100:00100000:0.0:1787826422.088685:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585272576 00000100:00100000:0.0:1787826422.088746:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 500 00000100:00100000:0.0:1787826422.088755:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768895b9c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:8435:x1874670585272576:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826422.088772:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826422.088869:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826422.088877:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768895b9c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:8435:x1874670585272576:12345-192.168.203.139@tcp:104: Request processed in 123us (318us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826422.088884:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 500 00010000:00010000:3.0:1787826422.089025:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826422.089033:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826422.089042:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976890fde400 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826422.089048:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1787826422.089067:0:42221:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787826422.089080:0:42221:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787826422.089122:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 2/0,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826422.089130:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826422.100573:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826422.100594:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000308885d1/0x8e7c99ee62ab1418 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x10b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b82e5c expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00000100:00100000:0.0:1787826422.107247:0:55654:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee1c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55654:1874670563947776:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826422.107279:0:55654:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab142d lrc: 4/0,1 mode: --/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b82e8d expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.107294:0:55654:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab142d lrc: 4/0,1 mode: --/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b82e8d expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.107303:0:55654:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab142d lrc: 4/0,1 mode: --/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b82e8d expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826422.114378:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826422.114512:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585272704 00000100:00100000:1.0:1787826422.114522:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585272704 00000100:00100000:1.0:1787826422.114579:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 501 00000100:00100000:1.0:1787826422.114588:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aec000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:6504:x1874670585272704:12345-192.168.203.139@tcp:105: 00010000:00010000:1.0:1787826422.114605:0:53061:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab142d lrc: 5/0,1 mode: --/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b82e8d expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826422.114618:0:53061:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab142d lrc: 5/0,1 mode: --/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b82e8d expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826422.114627:0:53061:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab142d lrc: 5/0,1 mode: --/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b82e8d expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826422.114639:0:53061:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab142d lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x22153cd396b82e8d expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826422.114648:0:53061:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab142d lrc: 6/0,1 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x22153cd396b82e8d expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826422.114664:0:53061:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab142d lrc: 6/0,1 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x22153cd396b82e8d expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826422.114700:0:53061:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff97688925da00) 00000100:00100000:1.0:1787826422.114769:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aec000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:6504:x1874670585272704:12345-192.168.203.139@tcp:105: Request processed in 182us (401us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826422.114776:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 501 00000020:00010000:0.0:1787826422.119281:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976890fb73f8 (ffff976890d10758/0) at cl_lock_release():245 00000020:00010000:0.0:1787826422.119285:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976890fb73f8 (ffff976890d10758/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826422.119291:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e554b0 (ffff97688386c098/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787826422.119303:0:55654:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab142d lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b82e8d expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.119318:0:55654:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab142d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82e8d expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787826422.119334:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976890fb73f8 (ffff976890d10758/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826422.119337:0:55654:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e554b0 (ffff976890d10758/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826422.119353:0:55654:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 27 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787826422.119367:0:55654:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8) current size 0, valid attrs 100101f0, mode 9768, uid 0, gid 0, new size 0, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826422.000000000, rc 0 00000080:00200080:0.0:1787826422.119408:0:55654:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=db2570 00010000:00010000:0.0:1787826422.119457:0:55654:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab1403 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0x22153cd396b82e39 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.119474:0:55654:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab1403 lrc: 0/0,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x85809400000000 nid: local remote: 0x22153cd396b82e39 expref: -99 pid: 55654 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826422.119491:0:55654:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889efb0c0 x1874670563945472/t8589935113(8589935113) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826437 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826422.119501:0:55654:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826422.119506:0:55654:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aed680 x1874670563948288/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 504/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826422.119526:0:55654:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55654:1874670563948288:192.168.203.139@tcp:35:lfs.0 00000100:00100000:0.0:1787826422.119616:0:55654:0:(client.c:2715:ptlrpc_set_wait()) set ffff976891c9a000 going to sleep for 11 seconds 00000100:00100000:0.0:1787826422.135336:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826422.135494:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585272832 00000100:00100000:1.0:1787826422.135504:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585272832 00000100:00100000:1.0:1787826422.135564:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 502 00000100:00100000:1.0:1787826422.135574:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aee940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585272832:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826422.135593:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfa00/0x8e7c99ee62ab13c4 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82db4 expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1787826422.135673:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826422.135680:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aee940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585272832:12345-192.168.203.139@tcp:104: Request processed in 108us (355us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826422.135688:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 502 00000100:00100000:1.0:1787826422.139485:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826422.139577:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585272960 00000100:00100000:0.0:1787826422.139587:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585272960 00000100:00100000:0.0:1787826422.139641:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 503 00000100:00100000:0.0:1787826422.139651:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0163480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585272960:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826422.139667:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab13af lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82d67 expref: -99 pid: 55646 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826422.139748:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826422.139755:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0163480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585272960:12345-192.168.203.139@tcp:104: Request processed in 106us (278us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826422.139762:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 503 00010000:00010000:3.0:1787826422.139855:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826422.139863:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab13af lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82d67 expref: -99 pid: 55646 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826422.139872:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925c400 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826422.139877:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab13af lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82d67 expref: -99 pid: 55646 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826422.139885:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab13af lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82d67 expref: -99 pid: 55646 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826422.139894:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab13af lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b82d67 expref: -99 pid: 55646 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1787826422.139918:0:42220:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x1d:0x0]: losing layout lock 00010000:00010000:3.0:1787826422.139955:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab13af lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82d67 expref: -99 pid: 55646 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826422.139963:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826422.139991:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab13af lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82d67 expref: -99 pid: 55646 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826422.139999:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab13af lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82d67 expref: -99 pid: 55646 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826422.140013:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826422.140015:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826422.140017:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826422.140025:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826422.140038:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563948672:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826422.157964:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563948672:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826422.159872:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826422.161742:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585273088 00000100:00100000:1.0:1787826422.161754:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585273088 00000100:00100000:1.0:1787826422.161817:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 504 00000100:00100000:1.0:1787826422.161846:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0161a40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585273088:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826422.161865:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab141f lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x38:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82e78 expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826422.161952:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826422.161958:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0161a40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585273088:12345-192.168.203.139@tcp:104: Request processed in 134us (2094us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826422.161965:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 504 00010000:00010000:3.0:1787826422.241322:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826422.250337:0:55656:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efad00 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55656:1874670563949696:192.168.203.139@tcp:101:getfattr.0 00010000:00010000:3.0:1787826422.250365:0:55656:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 4/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.250396:0:55656:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.250404:0:55656:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.250415:0:55656:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.250426:0:55656:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.250440:0:55656:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.250449:0:55656:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.250457:0:55656:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826422.250469:0:55656:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff976889efad00 x1874670563949696/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 584/824 e 0 to 0 dl 1787826438 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'getfattr.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826422.250479:0:55656:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826422.250492:0:55656:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.250501:0:55656:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826422.250508:0:55656:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1787826422.250559:0:55656:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x1d:0x0], updating i_size 0 00000080:00200000:3.0:1787826422.250563:0:55656:0:(namei.c:147:ll_iget()) got inode: [0x200000405:0x1d:0x0](ffff9768901a80c8): rc = 0 00000080:00200000:3.0:1787826422.250577:0:55656:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 28 -> 28 00000080:00010000:3.0:1787826422.250581:0:55656:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab143b 00000080:00010000:3.0:1787826422.250592:0:55656:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:3.0:1787826422.250596:0:55656:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab143b from it ffffb2d4421abb20 00010000:00010000:3.0:1787826422.250599:0:55656:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.250608:0:55656:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1787826422.250644:0:55656:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826422.250651:0:55656:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:3.0:1787826422.250672:0:55656:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826422.250681:0:55656:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.250692:0:55656:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826422.250699:0:55656:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826422.250704:0:55656:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:3.0:1787826422.250707:0:55656:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab143b 00000080:00010000:3.0:1787826422.250711:0:55656:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab143b from it ffffb2d4421abcb8 00010000:00010000:3.0:1787826422.250724:0:55656:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.250732:0:55656:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826422.250774:0:55656:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826422.250783:0:55656:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826422.250791:0:55656:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 227 lru_type: 1 00010000:00010000:3.0:1787826422.250799:0:55656:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 227 lru_type: 1 00000080:00200000:3.0:1787826422.250806:0:55656:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826422.250810:0:55656:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826422.250812:0:55656:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826422.250816:0:55656:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4421abbf8 00010000:00010000:3.0:1787826422.250820:0:55656:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 227 lru_type: 1 00010000:00010000:3.0:1787826422.250851:0:55656:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 228 lru_type: 1 00000080:00200000:3.0:1787826422.250860:0:55656:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826422.250867:0:55656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:3.0:1787826422.250871:0:55656:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787826422.250874:0:55656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=9 00000080:00200000:3.0:1787826422.250880:0:55656:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:3.0:1787826422.250885:0:55656:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200000:3.0:1787826422.250901:0:55656:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826422.250908:0:55656:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826422.250916:0:55656:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 228 lru_type: 1 00010000:00010000:3.0:1787826422.250924:0:55656:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 228 lru_type: 1 00000080:00200000:3.0:1787826422.250931:0:55656:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826422.250935:0:55656:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826422.250937:0:55656:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826422.250941:0:55656:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4421abbf8 00010000:00010000:3.0:1787826422.250943:0:55656:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 228 lru_type: 1 00010000:00010000:3.0:1787826422.250950:0:55656:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 229 lru_type: 1 00000080:00200000:3.0:1787826422.250959:0:55656:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826422.250962:0:55656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:3.0:1787826422.250965:0:55656:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787826422.250968:0:55656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=9 00000080:00200000:3.0:1787826422.250974:0:55656:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:3.0:1787826422.250977:0:55656:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200000:3.0:1787826422.312381:0:55660:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826422.316417:0:55660:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826422.316437:0:55660:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 229 lru_type: 1 00010000:00010000:3.0:1787826422.316453:0:55660:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 229 lru_type: 1 00000080:00200000:3.0:1787826422.316468:0:55660:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826422.316476:0:55660:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826422.316480:0:55660:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826422.316488:0:55660:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44213bbf8 00010000:00010000:3.0:1787826422.316492:0:55660:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 229 lru_type: 1 00010000:00010000:3.0:1787826422.316501:0:55660:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 230 lru_type: 1 00000080:00200000:3.0:1787826422.316517:0:55660:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826422.316525:0:55660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=27 00000080:00200000:3.0:1787826422.316532:0:55660:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787826422.316536:0:55660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=267 00000080:00200000:3.0:1787826422.316595:0:55660:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826422.316604:0:55660:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826422.316611:0:55660:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 230 lru_type: 1 00010000:00010000:3.0:1787826422.316620:0:55660:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 230 lru_type: 1 00000080:00200000:3.0:1787826422.316625:0:55660:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826422.316629:0:55660:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826422.316632:0:55660:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826422.316636:0:55660:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44213bbf8 00010000:00010000:3.0:1787826422.316639:0:55660:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 230 lru_type: 1 00010000:00010000:3.0:1787826422.316647:0:55660:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 231 lru_type: 1 00000080:00200000:3.0:1787826422.316656:0:55660:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826422.316660:0:55660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=267 00000080:00200000:3.0:1787826422.316664:0:55660:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 44 00000080:00200008:3.0:1787826422.316669:0:55660:0:(file.c:1090:ll_file_open()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00000080:00200080:3.0:1787826422.316705:0:55660:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8) cmd=c0086915 arg=c603d0 00000100:00100000:3.0:1787826422.316985:0:55660:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55660:1874670563949824:192.168.203.139@tcp:34:lfs.0 00000100:00100000:3.0:1787826422.317494:0:55660:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25300 going to sleep for 11 seconds 00000100:00100000:3.0:1787826422.326444:0:55660:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55660:1874670563949824:192.168.203.139@tcp:34:lfs.0 00000080:00200008:3.0:1787826422.326969:0:55660:0:(file.c:479:ll_file_release()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00010000:00010000:3.0:1787826422.327002:0:55660:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9600/0x8e7c99ee62ab0c1c lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b818c9 expref: -99 pid: 55483 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:0.0:1787826422.357248:0:55662:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826422.357281:0:55662:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826422.357299:0:55662:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 231 lru_type: 1 00010000:00010000:0.0:1787826422.357315:0:55662:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 231 lru_type: 1 00000080:00200000:0.0:1787826422.357323:0:55662:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826422.357330:0:55662:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826422.357334:0:55662:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826422.357342:0:55662:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d442183bf8 00010000:00010000:0.0:1787826422.357345:0:55662:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 231 lru_type: 1 00010000:00010000:0.0:1787826422.357354:0:55662:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 232 lru_type: 1 00000080:00200000:0.0:1787826422.357372:0:55662:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826422.357380:0:55662:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:0.0:1787826422.357385:0:55662:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826422.357390:0:55662:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:0.0:1787826422.357393:0:55662:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 44 00000080:00200008:0.0:1787826422.357399:0:55662:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154400 00000002:00010000:0.0:1787826422.357414:0:55662:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114401 00010000:00010000:0.0:1787826422.357465:0:55662:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1442 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 55662 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.357475:0:55662:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1442 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55662 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826422.357487:0:55662:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826422.357499:0:55662:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aec3c0 x1874670563950080/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826422.357510:0:55662:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1442 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55662 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826422.357532:0:55662:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55662:1874670563950080:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826422.358986:0:55662:0:(client.c:2715:ptlrpc_set_wait()) set ffff976891c9a540 going to sleep for 11 seconds 00000100:00100000:1.0:1787826422.368124:0:55662:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55662:1874670563950080:192.168.203.139@tcp:101:lfs.0 00010000:00010000:1.0:1787826422.368151:0:55662:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1442 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82ee8 expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826422.368168:0:55662:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1442 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82ee8 expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826422.368179:0:55662:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1442 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82ee8 expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787826422.368193:0:55662:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff976882aec3c0 x1874670563950080/t8589935122(8589935122) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826438 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1787826422.368215:0:55662:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aec3c0 x1874670563950080/t8589935122(8589935122) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826438 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787826422.368227:0:55662:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1442 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82ee8 expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787826422.368239:0:55662:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1442 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82ee8 expref: -99 pid: 55662 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826422.368252:0:55662:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:1.0:1787826422.368258:0:55662:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1787826422.368302:0:55662:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x1d:0x0], updating i_size 0 00000080:00200000:1.0:1787826422.368315:0:55662:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 28 -> 28 00000080:00010000:1.0:1787826422.368320:0:55662:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1442 00000080:00010000:1.0:1787826422.368326:0:55662:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1442 from it ffffb2d442183bf0 00010000:00010000:1.0:1787826422.368329:0:55662:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1442 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82ee8 expref: -99 pid: 55662 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826422.368338:0:55662:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1442 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82ee8 expref: -99 pid: 55662 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000002:00100000:1.0:1787826422.368365:0:55662:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aec3c0 x1874670563950080/t8589935122(8589935122) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826438 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1787826422.368498:0:55662:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:1.0:1787826422.368506:0:55662:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:1.0:1787826422.368534:0:55662:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1787826422.368541:0:55662:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:1.0:1787826422.368563:0:55662:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826422.368574:0:55662:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826422.368591:0:55662:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787826422.368599:0:55662:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826422.368605:0:55662:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:1.0:1787826422.368608:0:55662:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab143b 00000080:00010000:1.0:1787826422.368613:0:55662:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab143b from it ffffb2d442183cd8 00010000:00010000:1.0:1787826422.368616:0:55662:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826422.368623:0:55662:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:1.0:1787826422.368666:0:55662:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154000 00010000:00010000:1.0:1787826422.368676:0:55662:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1442 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82ee8 expref: -99 pid: 55662 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787826422.369051:0:55662:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826422.373750:0:55662:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826422.373775:0:55662:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 232 lru_type: 1 00010000:00010000:0.0:1787826422.373792:0:55662:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 232 lru_type: 1 00000080:00200000:0.0:1787826422.373801:0:55662:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826422.374025:0:55662:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826422.374031:0:55662:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826422.374038:0:55662:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d442183bf8 00010000:00010000:0.0:1787826422.374042:0:55662:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 232 lru_type: 1 00010000:00010000:0.0:1787826422.374051:0:55662:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 233 lru_type: 1 00000080:00200000:0.0:1787826422.374072:0:55662:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826422.374082:0:55662:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:0.0:1787826422.374088:0:55662:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826422.374092:0:55662:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:0.0:1787826422.374096:0:55662:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 44 00000080:00200008:0.0:1787826422.374103:0:55662:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154400 00000080:00200000:0.0:1787826422.374173:0:55662:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:0.0:1787826422.374180:0:55662:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:0.0:1787826422.374222:0:55662:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826422.374228:0:55662:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:0.0:1787826422.374245:0:55662:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826422.374255:0:55662:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826422.374264:0:55662:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826422.374271:0:55662:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826422.374276:0:55662:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:0.0:1787826422.374279:0:55662:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab143b 00000080:00010000:0.0:1787826422.374284:0:55662:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab143b from it ffffb2d442183cd8 00010000:00010000:0.0:1787826422.374287:0:55662:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826422.374294:0:55662:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200008:0.0:1787826422.374337:0:55662:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154000 00010000:00010000:0.0:1787826422.374347:0:55662:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1442 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82ee8 expref: -99 pid: 55662 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826422.374387:0:55662:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826422.374395:0:55662:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826422.374401:0:55662:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 233 lru_type: 1 00010000:00010000:0.0:1787826422.374409:0:55662:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 233 lru_type: 1 00000080:00200000:0.0:1787826422.374416:0:55662:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826422.374420:0:55662:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826422.374422:0:55662:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826422.374426:0:55662:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d442183bf8 00010000:00010000:0.0:1787826422.374429:0:55662:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 233 lru_type: 1 00010000:00010000:0.0:1787826422.374435:0:55662:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 234 lru_type: 1 00000080:00200000:0.0:1787826422.374442:0:55662:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826422.374446:0:55662:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:0.0:1787826422.374450:0:55662:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826422.374453:0:55662:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:0.0:1787826422.374456:0:55662:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 46 00000080:00200008:0.0:1787826422.374459:0:55662:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154402 00000002:00010000:0.0:1787826422.374469:0:55662:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114403 00010000:00010000:0.0:1787826422.374519:0:55662:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1449 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 55662 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.374526:0:55662:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1449 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55662 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826422.374537:0:55662:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826422.374551:0:55662:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aecf00 x1874670563950336/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826422.374561:0:55662:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1449 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55662 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826422.375271:0:55662:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aecf00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55662:1874670563950336:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826422.375516:0:55662:0:(client.c:2715:ptlrpc_set_wait()) set ffff976891c9a540 going to sleep for 11 seconds 00000100:00100000:0.0:1787826422.390021:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826422.390118:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585273600 00000100:00100000:1.0:1787826422.390128:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585273600 00000100:00100000:1.0:1787826422.390183:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 505 00000100:00100000:1.0:1787826422.390194:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aef480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6518:x1874670585273600:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826422.390221:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787826422.390394:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826422.390402:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aef480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6518:x1874670585273600:12345-192.168.203.139@tcp:104: Request processed in 210us (390us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826422.390410:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 505 00010000:00010000:3.0:1787826422.392730:0:49796:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1442 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82ee8 expref: -99 pid: 55662 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826422.392787:0:49796:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 2/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826422.392794:0:49796:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826422.393196:0:49796:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 1/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826422.393215:0:49796:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab143b lrc: 0/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82ec5 expref: -99 pid: 55656 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826422.393232:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826422.393234:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826422.393236:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826422.407622:0:55662:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aecf00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55662:1874670563950336:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826422.407650:0:55662:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode CW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1449 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82efd expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.407663:0:55662:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1449 lrc: 4/1,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82efd expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.407675:0:55662:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1449 lrc: 4/1,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82efd expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.407691:0:55662:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1449 lrc: 4/1,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82efd expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.407705:0:55662:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1449 lrc: 5/1,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82efd expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.407713:0:55662:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1449 lrc: 5/1,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82efd expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.407720:0:55662:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1449 lrc: 4/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82efd expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826422.407732:0:55662:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff976882aecf00 x1874670563950336/t8589935123(8589935123) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826438 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826422.407744:0:55662:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aecf00 x1874670563950336/t8589935123(8589935123) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826438 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826422.407757:0:55662:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1449 lrc: 3/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82efd expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.407771:0:55662:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 8 res 8589935621/29 (0 0) 00000080:00200000:0.0:1787826422.407776:0:55662:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826422.407871:0:55662:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x1d:0x0], updating i_size 0 00000080:00010000:0.0:1787826422.407877:0:55662:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1449 00000080:00010000:0.0:1787826422.407884:0:55662:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1449 from it ffffb2d442183bf0 00010000:00010000:0.0:1787826422.407888:0:55662:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1449 lrc: 3/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82efd expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.407897:0:55662:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1449 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82efd expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:0.0:1787826422.407926:0:55662:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aecf00 x1874670563950336/t8589935123(8589935123) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826438 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:0.0:1787826422.408059:0:55662:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=800466af arg=7ffc86b0e70c 00000080:00200080:0.0:1787826422.408074:0:55662:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=80086601 arg=7ffc86b0e6a8 00000100:00100000:0.0:1787826422.408115:0:55662:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55662:1874670563950592:192.168.203.139@tcp:33:lfs.0 00000100:00100000:0.0:1787826422.408197:0:55662:0:(client.c:2715:ptlrpc_set_wait()) set ffff976891c9a540 going to sleep for 11 seconds 00000100:00100000:0.0:1787826422.416113:0:55662:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55662:1874670563950592:192.168.203.139@tcp:33:lfs.0 00000080:00200080:0.0:1787826422.416158:0:55662:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=7ffc86b0e694 00000002:00010000:0.0:1787826422.416185:0:55662:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 01004040000002 00010000:00010000:0.0:1787826422.416216:0:55662:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1449 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x22153cd396b82efd expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826422.416248:0:55662:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1442 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b82ee8 expref: -99 pid: 55662 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826422.416264:0:55662:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1434 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82ea2 expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000002:00100000:0.0:1787826422.416306:0:55662:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976882aec3c0 x1874670563950080/t8589935122(8589935122) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826438 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826422.416316:0:55662:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826422.416322:0:55662:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aef840 x1874670563950848/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826422.416344:0:55662:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55662:1874670563950848:192.168.203.139@tcp:35:lfs.0 00000100:00100000:0.0:1787826422.416433:0:55662:0:(client.c:2715:ptlrpc_set_wait()) set ffff976891c9a540 going to sleep for 11 seconds 00000100:00100000:0.0:1787826422.424763:0:55662:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55662:1874670563950848:192.168.203.139@tcp:35:lfs.0 00000002:00100000:0.0:1787826422.424801:0:55662:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976882aec3c0 x1874670563950080/t8589935122(8589935122) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826438 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1787826422.424930:0:55662:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1449 lrc: 1/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b82efd expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826422.424939:0:55662:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1442 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b82ee8 expref: -99 pid: 55662 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826422.424945:0:55662:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787826422.424949:0:55662:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1449 lrc: 0/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b82efd expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826422.424961:0:55662:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1442 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b82ee8 expref: -99 pid: 55662 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826422.424987:0:55662:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1450 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 55662 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.424994:0:55662:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x81000000001000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1450 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 55662 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826422.425004:0:55662:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826422.425010:0:55662:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aef840 x1874670563951104/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 976/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826422.425021:0:55662:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1450 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x81000000000000 pid: 55662 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826422.425041:0:55662:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55662:1874670563951104:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826422.425214:0:55662:0:(client.c:2715:ptlrpc_set_wait()) set ffff976891c9a540 going to sleep for 11 seconds 00000100:00100000:0.0:1787826422.428115:0:55662:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55662:1874670563951104:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826422.428141:0:55662:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1450 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b82f12 expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.428154:0:55662:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1450 lrc: 4/1,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b82f12 expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.428166:0:55662:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1450 lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b82f12 expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.428177:0:55662:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1450 lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b82f12 expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.428190:0:55662:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1450 lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b82f12 expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.428197:0:55662:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1450 lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b82f12 expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.428216:0:55662:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1450 lrc: 4/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b82f12 expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826422.428233:0:55662:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x600002b, status=0 req@ffff976882aef840 x1874670563951104/t8589935125(8589935125) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 976/848 e 0 to 0 dl 1787826438 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826422.428245:0:55662:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aef840 x1874670563951104/t8589935125(8589935125) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 976/848 e 0 to 0 dl 1787826438 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826422.428260:0:55662:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1450 lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b82f12 expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.428275:0:55662:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 2 res 8589935621/29 (0 0) 00000080:00200000:0.0:1787826422.428282:0:55662:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000002:00100000:0.0:1787826422.428302:0:55662:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aef840 x1874670563951104/t8589935125(8589935125) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 768/848 e 0 to 0 dl 1787826438 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787826422.428310:0:55662:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1450 00000080:00010000:0.0:1787826422.428316:0:55662:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1450 from it ffffb2d442183cc0 00010000:00010000:0.0:1787826422.428320:0:55662:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1450 lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b82f12 expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.428327:0:55662:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1450 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b82f12 expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200080:0.0:1787826422.428543:0:55662:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=acd2e0 00010000:00010000:0.0:1787826422.428586:0:55662:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1450 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0x22153cd396b82f12 expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.428601:0:55662:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1450 lrc: 0/0,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x85809400000000 nid: local remote: 0x22153cd396b82f12 expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826422.428621:0:55662:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976882aef840 x1874670563951104/t8589935125(8589935125) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 768/848 e 0 to 0 dl 1787826438 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826422.428630:0:55662:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826422.428635:0:55662:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aec3c0 x1874670563951360/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 504/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826422.428655:0:55662:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55662:1874670563951360:192.168.203.139@tcp:35:lfs.0 00000100:00100000:0.0:1787826422.428734:0:55662:0:(client.c:2715:ptlrpc_set_wait()) set ffff976891c9a540 going to sleep for 11 seconds 00000100:00100000:1.0:1787826422.434103:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826422.434139:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585273728 00000100:00100000:1.0:1787826422.434148:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585273728 00000100:00100000:1.0:1787826422.434197:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 506 00000100:00100000:1.0:1787826422.434219:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0160b40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6521:x1874670585273728:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826422.434240:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1434 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82ea2 expref: -99 pid: 55654 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787826422.434327:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826422.434334:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0160b40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6521:x1874670585273728:12345-192.168.203.139@tcp:104: Request processed in 128us (240us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826422.434342:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 506 00000100:00100000:1.0:1787826422.445137:0:55662:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55662:1874670563951360:192.168.203.139@tcp:35:lfs.0 00000080:00020000:1.0:1787826422.445153:0:55662:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9768877a4800: inode [0x200000405:0x1d:0x0] mdc close failed: rc = -22 00000002:00100000:1.0:1787826422.445171:0:55662:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976882aef840 x1874670563951104/t8589935125(8589935125) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 768/848 e 0 to 0 dl 1787826438 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1787826422.445219:0:55662:0:(file.c:7076:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000002:00010000:1.0:1787826422.445230:0:55662:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: layout flags 0 00010000:00010000:1.0:1787826422.445281:0:55662:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde200/0x8e7c99ee62ab1457 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55662 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826422.445292:0:55662:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde200/0x8e7c99ee62ab1457 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55662 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826422.445302:0:55662:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde200/0x8e7c99ee62ab1457 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55662 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826422.445321:0:55662:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0162d00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55662:1874670563951616:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826422.447023:0:55662:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3480 going to sleep for 11 seconds 00000100:00100000:0.0:1787826422.477219:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826422.477327:0:42211:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1787826422.477347:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585273856 00000100:00100000:1.0:1787826422.477355:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585273856 00000100:00100000:1.0:1787826422.477410:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 507 00000100:00100000:1.0:1787826422.477420:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768895bb050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6522:x1874670585273856:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826422.477437:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde200/0x8e7c99ee62ab1457 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82f27 expref: -99 pid: 55662 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826422.477507:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826422.477513:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768895bb050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6522:x1874670585273856:12345-192.168.203.139@tcp:104: Request processed in 95us (313us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826422.477521:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 507 00010000:00010000:3.0:1787826422.477581:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826422.477601:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde200/0x8e7c99ee62ab1457 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82f27 expref: -99 pid: 55662 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826422.477615:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976890fde200 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826422.477621:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde200/0x8e7c99ee62ab1457 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82f27 expref: -99 pid: 55662 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826422.477631:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde200/0x8e7c99ee62ab1457 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82f27 expref: -99 pid: 55662 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826422.477643:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde200/0x8e7c99ee62ab1457 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b82f27 expref: -99 pid: 55662 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826422.477675:0:42220:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x1d:0x0]: losing layout lock 00010000:00010000:3.0:1787826422.477747:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde200/0x8e7c99ee62ab1457 lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82f27 expref: -99 pid: 55662 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826422.477755:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826422.478098:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde200/0x8e7c99ee62ab1457 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82f27 expref: -99 pid: 55662 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826422.478118:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde200/0x8e7c99ee62ab1457 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82f27 expref: -99 pid: 55662 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826422.478153:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826422.478156:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826422.478160:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826422.485240:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:3.0:1787826422.485417:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563952512:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826422.489170:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585273984 00000100:00100000:0.0:1787826422.489186:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585273984 00000100:00100000:0.0:1787826422.489261:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 508 00000100:00100000:0.0:1787826422.489271:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aef0c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6522:x1874670585273984:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826422.489289:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab145e lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82f35 expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826422.489384:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826422.489391:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aef0c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6522:x1874670585273984:12345-192.168.203.139@tcp:104: Request processed in 123us (4162us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826422.489398:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 508 00010000:00010000:3.0:1787826422.489562:0:49796:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826422.489570:0:49796:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab145e lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82f35 expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.489579:0:49796:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514200 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826422.489584:0:49796:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab145e lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x39:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82f35 expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.489592:0:49796:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab145e lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x39:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82f35 expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.489674:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab145e lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x39:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b82f35 expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826422.489699:0:49796:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x39:0x0]: losing layout lock 00010000:00010000:3.0:1787826422.489741:0:49796:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab145e lrc: 2/0,0 mode: --/PR res: [0x200000405:0x39:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82f35 expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.489748:0:49796:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826422.489776:0:49796:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab145e lrc: 1/0,0 mode: --/PR res: [0x200000405:0x39:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82f35 expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.489784:0:49796:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab145e lrc: 0/0,0 mode: --/PR res: [0x200000405:0x39:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82f35 expref: -99 pid: 55662 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.489804:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826422.489815:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826422.489818:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826422.489844:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826422.489858:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563952640:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826422.492427:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563952640:192.168.203.139@tcp:103:ldlm_bl.0 00000002:00100000:3.0:1787826422.520964:0:55662:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889ef9680 x1874670563951872/t8589935127(8589935127) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826438 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826422.520982:0:55662:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1787826422.521058:0:55662:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efc780 x1874670563953024/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826422.521087:0:55662:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55662:1874670563953024:192.168.203.139@tcp:35:lfs.0 00000100:00100000:3.0:1787826422.521249:0:55662:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25300 going to sleep for 11 seconds 00000100:00100000:3.0:1787826422.553365:0:55662:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55662:1874670563953024:192.168.203.139@tcp:35:lfs.0 00000002:00100000:3.0:1787826422.553407:0:55662:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889ef9680 x1874670563951872/t8589935127(8589935127) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826438 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1787826422.553481:0:55662:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154002 00000002:00100000:3.0:1787826422.553535:0:55662:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976882aecf00 x1874670563950336/t8589935123(8589935123) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826438 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826422.553545:0:55662:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1787826422.553549:0:55662:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efda40 x1874670563953280/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826422.553573:0:55662:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efda40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55662:1874670563953280:192.168.203.139@tcp:35:lfs.0 00000080:00200000:3.0:1787826422.572021:0:55664:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826422.572639:0:55664:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826422.572661:0:55664:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 235 lru_type: 1 00010000:00010000:3.0:1787826422.572677:0:55664:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 235 lru_type: 1 00000080:00200000:3.0:1787826422.572685:0:55664:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826422.572694:0:55664:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826422.572698:0:55664:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826422.572704:0:55664:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44219bb58 00010000:00010000:3.0:1787826422.572708:0:55664:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 235 lru_type: 1 00010000:00010000:3.0:1787826422.572718:0:55664:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 236 lru_type: 1 00000080:00200000:3.0:1787826422.572734:0:55664:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826422.572742:0:55664:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=24 00000080:00200000:3.0:1787826422.572747:0:55664:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787826422.572759:0:55664:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), flags=8 00000080:00200000:3.0:1787826422.572763:0:55664:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=getattr 00000002:00010000:3.0:1787826422.572778:0:55664:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f60b.sanity-flr,[0x0:0x0:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 01 00010000:00010000:3.0:1787826422.573692:0:55664:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1473 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55664 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.573705:0:55664:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1473 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55664 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.573718:0:55664:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1473 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55664 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826422.573744:0:55664:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efda40 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55664:1874670563953408:192.168.203.139@tcp:101:getfattr.0 00000100:00100000:3.0:1787826422.573947:0:55664:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25900 going to sleep for 11 seconds 00000100:00100000:3.0:1787826422.591289:0:55664:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efda40 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55664:1874670563953408:192.168.203.139@tcp:101:getfattr.0 00010000:00010000:3.0:1787826422.591319:0:55664:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1473 lrc: 4/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82f89 expref: -99 pid: 55664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.591345:0:55664:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1473 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82f89 expref: -99 pid: 55664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.591353:0:55664:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1473 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82f89 expref: -99 pid: 55664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.591363:0:55664:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1473 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82f89 expref: -99 pid: 55664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.591375:0:55664:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1473 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82f89 expref: -99 pid: 55664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.591388:0:55664:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1473 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82f89 expref: -99 pid: 55664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.591402:0:55664:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1473 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82f89 expref: -99 pid: 55664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.591410:0:55664:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1473 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82f89 expref: -99 pid: 55664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826422.591421:0:55664:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff976889efda40 x1874670563953408/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 584/720 e 0 to 0 dl 1787826438 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'getfattr.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826422.591434:0:55664:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1473 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82f89 expref: -99 pid: 55664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826422.591448:0:55664:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1473 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82f89 expref: -99 pid: 55664 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.591459:0:55664:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826422.591465:0:55664:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1787826422.591510:0:55664:0:(namei.c:147:ll_iget()) got inode: [0x200000405:0x1d:0x0](ffff9768901a80c8): rc = 0 00000080:00200000:3.0:1787826422.591524:0:55664:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 29 -> 29 00000080:00010000:3.0:1787826422.591529:0:55664:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1473 00000080:00010000:3.0:1787826422.591539:0:55664:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:3.0:1787826422.591543:0:55664:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1473 from it ffffb2d44219bb20 00010000:00010000:3.0:1787826422.591547:0:55664:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1473 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82f89 expref: -99 pid: 55664 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.591556:0:55664:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1473 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82f89 expref: -99 pid: 55664 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1787826422.591660:0:55664:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826422.591668:0:55664:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:3.0:1787826422.591692:0:55664:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826422.591702:0:55664:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1473 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82f89 expref: -99 pid: 55664 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.591712:0:55664:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1473 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82f89 expref: -99 pid: 55664 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826422.591720:0:55664:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826422.591725:0:55664:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:3.0:1787826422.591728:0:55664:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1473 00000080:00010000:3.0:1787826422.591732:0:55664:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1473 from it ffffb2d44219bcb8 00010000:00010000:3.0:1787826422.591736:0:55664:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1473 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82f89 expref: -99 pid: 55664 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.591743:0:55664:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1473 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82f89 expref: -99 pid: 55664 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826422.591763:0:55664:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 27 restore needed 0 00000001:00010000:3.0:1787826422.591774:0:55664:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:3.0:1787826422.591792:0:55664:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1787826422.591795:0:55664:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff9768837ffcb0, osc lock ffff9768984ee4e0, flags 0x400001002 00010000:00010000:3.0:1787826422.591812:0:55664:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.591872:0:55664:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.591882:0:55664:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.591890:0:55664:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826422.591928:0:55664:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab147a lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55664 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.591937:0:55664:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab147a lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55664 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787826422.591966:0:55664:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1787826422.591986:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563953536:192.168.203.139@tcp:101:getfattr.0 00000100:00100000:0.0:1787826422.594525:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826422.594613:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585274240 00000100:00100000:1.0:1787826422.594625:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585274240 00000100:00100000:1.0:1787826422.595042:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 509 00000100:00100000:1.0:1787826422.595054:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aeda40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30696:x1874670585274240:12345-192.168.203.139@tcp:106: 00010000:00010000:1.0:1787826422.595070:0:42210:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826422.595083:0:42210:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:1.0:1787826422.595110:0:42210:0:(cl_object.c:322:cl_object_glimpse()) header@00000000ca1bb7a3[0x110, 1, [0x200000405:0x38:0x0] hash] 00000020:00010000:1.0:1787826422.595113:0:42210:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826421 blocks: 0 00000020:00010000:1.0:1787826422.595117:0:42210:0:(cl_object.c:322:cl_object_glimpse()) header@0000000001146bd0[0x116, 2, [0x2c0000401:0x10c:0x0]] 00000020:00010000:1.0:1787826422.595119:0:42210:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826421 blocks: 0 00000100:00100000:1.0:1787826422.595179:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aeda40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30696:x1874670585274240:12345-192.168.203.139@tcp:106: Request processed in 129us (664us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826422.595187:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 509 00010000:00010000:3.0:1787826422.599418:0:2403:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab147a lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55664 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.599441:0:2403:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab147a lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55664 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.599455:0:2403:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab147a lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55664 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.599464:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab147a lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55664 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.599473:0:2403:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab147a lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55664 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826422.599488:0:2403:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:3.0:1787826422.599497:0:2403:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826422.599516:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab147a lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55664 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.599533:0:2403:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab147a lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55664 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.599541:0:2403:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab147a lrc: 4/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55664 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.599549:0:2403:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976888fb8a00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826422.599555:0:2403:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab147a lrc: 5/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55664 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.599566:0:2403:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826422.599574:0:2403:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab147a lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55664 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.599587:0:2403:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab147a lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55664 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826422.599609:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563953536:192.168.203.139@tcp:101:getfattr.0 00000080:00200000:1.0:1787826422.599711:0:55664:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:1.0:1787826422.599719:0:55664:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9768983a3160 (ffff976883b9dea8/0) at cl_lock_release():245 00000020:00010000:1.0:1787826422.599722:0:55664:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768983a3160 (ffff976883b9dea8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1787826422.599728:0:55664:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768837ffcb0 (ffff976890d10638/1) at cl_lock_cancel():142 00000020:00010000:1.0:1787826422.599732:0:55664:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768983a3160 (ffff976883b9dea8/0) at cl_lock_fini():72 00000020:00010000:1.0:1787826422.599735:0:55664:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768837ffcb0 (ffff976883b9dea8/1) at cl_lock_fini():72 00000080:00200000:1.0:1787826422.599776:0:55664:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 29 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1787826422.600084:0:55664:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826422.600112:0:55664:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826422.600129:0:55664:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 236 lru_type: 1 00010000:00010000:1.0:1787826422.600142:0:55664:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 236 lru_type: 1 00000080:00200000:1.0:1787826422.600150:0:55664:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826422.600157:0:55664:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826422.600161:0:55664:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826422.600180:0:55664:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44219bbf8 00010000:00010000:1.0:1787826422.600185:0:55664:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 236 lru_type: 1 00010000:00010000:1.0:1787826422.600194:0:55664:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 237 lru_type: 1 00000080:00200000:1.0:1787826422.600281:0:55664:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826422.600287:0:55664:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:1.0:1787826422.600293:0:55664:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826422.600297:0:55664:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=9 00000080:00200000:1.0:1787826422.600307:0:55664:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:1.0:1787826422.600312:0:55664:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200000:1.0:1787826422.600332:0:55664:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826422.600340:0:55664:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826422.600348:0:55664:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 237 lru_type: 1 00010000:00010000:1.0:1787826422.600357:0:55664:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 237 lru_type: 1 00000080:00200000:1.0:1787826422.600363:0:55664:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826422.600368:0:55664:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826422.600370:0:55664:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826422.600375:0:55664:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44219bbf8 00010000:00010000:1.0:1787826422.600378:0:55664:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 237 lru_type: 1 00010000:00010000:1.0:1787826422.600386:0:55664:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 238 lru_type: 1 00000080:00200000:1.0:1787826422.600394:0:55664:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826422.600398:0:55664:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:1.0:1787826422.600402:0:55664:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826422.600405:0:55664:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=9 00000080:00200000:1.0:1787826422.600413:0:55664:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:1.0:1787826422.600416:0:55664:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200000:1.0:1787826422.640115:0:55668:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826422.640148:0:55668:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826422.640167:0:55668:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 238 lru_type: 1 00010000:00010000:1.0:1787826422.640338:0:55668:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 238 lru_type: 1 00000080:00200000:1.0:1787826422.640348:0:55668:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826422.640355:0:55668:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826422.640359:0:55668:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826422.640366:0:55668:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44213bbf8 00010000:00010000:1.0:1787826422.640370:0:55668:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 238 lru_type: 1 00010000:00010000:1.0:1787826422.640379:0:55668:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 239 lru_type: 1 00000080:00200000:1.0:1787826422.640395:0:55668:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826422.640402:0:55668:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=27 00000080:00200000:1.0:1787826422.640408:0:55668:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826422.640412:0:55668:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=267 00000080:00200000:1.0:1787826422.640439:0:55668:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826422.640447:0:55668:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826422.640454:0:55668:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 239 lru_type: 1 00010000:00010000:1.0:1787826422.640463:0:55668:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 239 lru_type: 1 00000080:00200000:1.0:1787826422.640469:0:55668:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826422.640473:0:55668:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826422.640476:0:55668:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826422.640480:0:55668:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44213bbf8 00010000:00010000:1.0:1787826422.640483:0:55668:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 239 lru_type: 1 00010000:00010000:1.0:1787826422.640490:0:55668:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 240 lru_type: 1 00000080:00200000:1.0:1787826422.640499:0:55668:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826422.640502:0:55668:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=267 00000080:00200000:1.0:1787826422.640505:0:55668:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 44 00000080:00200008:1.0:1787826422.640511:0:55668:0:(file.c:1090:ll_file_open()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00000080:00200080:1.0:1787826422.640537:0:55668:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8) cmd=c0086915 arg=8b23d0 00000100:00100000:1.0:1787826422.640610:0:55668:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0163480 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55668:1874670563953664:192.168.203.139@tcp:34:lfs.0 00000100:00100000:1.0:1787826422.640865:0:55668:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3c00 going to sleep for 11 seconds 00000100:00100000:3.0:1787826422.645314:0:55668:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0163480 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55668:1874670563953664:192.168.203.139@tcp:34:lfs.0 00000080:00200008:3.0:1787826422.645505:0:55668:0:(file.c:479:ll_file_release()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00010000:00010000:3.0:1787826422.645536:0:55668:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9600/0x8e7c99ee62ab0c1c lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b818c9 expref: -99 pid: 55483 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:0.0:1787826422.657355:0:55670:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826422.657386:0:55670:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826422.657406:0:55670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 240 lru_type: 1 00010000:00010000:0.0:1787826422.657423:0:55670:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 240 lru_type: 1 00000080:00200000:0.0:1787826422.657431:0:55670:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826422.657439:0:55670:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826422.657442:0:55670:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826422.657450:0:55670:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4421c3bf8 00010000:00010000:0.0:1787826422.657454:0:55670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 240 lru_type: 1 00010000:00010000:0.0:1787826422.657463:0:55670:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 241 lru_type: 1 00000080:00200000:0.0:1787826422.657478:0:55670:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826422.657485:0:55670:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:0.0:1787826422.657491:0:55670:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826422.657495:0:55670:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:0.0:1787826422.657499:0:55670:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 44 00000080:00200008:0.0:1787826422.657504:0:55670:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154400 00000002:00010000:0.0:1787826422.657519:0:55670:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114401 00010000:00010000:0.0:1787826422.657567:0:55670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1481 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 55670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.657575:0:55670:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1481 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55670 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826422.657587:0:55670:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826422.657600:0:55670:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aec780 x1874670563953920/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826422.657610:0:55670:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1481 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55670 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826422.657633:0:55670:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55670:1874670563953920:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826422.657798:0:55670:0:(client.c:2715:ptlrpc_set_wait()) set ffff976891c9a600 going to sleep for 11 seconds 00000100:00100000:0.0:1787826422.683707:0:55670:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55670:1874670563953920:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826422.683741:0:55670:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1481 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82fb3 expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.683758:0:55670:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1481 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b82fb3 expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.683771:0:55670:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1481 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82fb3 expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826422.683786:0:55670:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff976882aec780 x1874670563953920/t8589935135(8589935135) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826438 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826422.683800:0:55670:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aec780 x1874670563953920/t8589935135(8589935135) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826438 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826422.683812:0:55670:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1481 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82fb3 expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826422.683876:0:55670:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1481 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b82fb3 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.683900:0:55670:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:0.0:1787826422.683906:0:55670:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826422.683957:0:55670:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 29 -> 29 00000080:00010000:0.0:1787826422.683963:0:55670:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1481 00000080:00010000:0.0:1787826422.683969:0:55670:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1481 from it ffffb2d4421c3bf0 00010000:00010000:0.0:1787826422.683973:0:55670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1481 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82fb3 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.683983:0:55670:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1481 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82fb3 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000002:00100000:0.0:1787826422.684011:0:55670:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aec780 x1874670563953920/t8589935135(8589935135) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826438 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826422.684150:0:55670:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:0.0:1787826422.684158:0:55670:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:0.0:1787826422.684187:0:55670:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826422.684194:0:55670:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:0.0:1787826422.684225:0:55670:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826422.684236:0:55670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1473 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82f89 expref: -99 pid: 55664 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826422.684247:0:55670:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1473 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82f89 expref: -99 pid: 55664 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826422.684255:0:55670:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826422.684261:0:55670:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:0.0:1787826422.684264:0:55670:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1473 00000080:00010000:0.0:1787826422.684268:0:55670:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1473 from it ffffb2d4421c3cd8 00010000:00010000:0.0:1787826422.684271:0:55670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1473 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b82f89 expref: -99 pid: 55664 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826422.684278:0:55670:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1473 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b82f89 expref: -99 pid: 55664 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826422.684295:0:55670:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 27 restore needed 0 00000001:00010000:0.0:1787826422.684305:0:55670:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:0.0:1787826422.684324:0:55670:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826422.684327:0:55670:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e554b0, osc lock ffff9768849a3860, flags 0x400001002 00010000:00010000:0.0:1787826422.684342:0:55670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826422.684353:0:55670:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826422.684362:0:55670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826422.684370:0:55670:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826422.684405:0:55670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab1488 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.684419:0:55670:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab1488 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826422.684446:0:55670:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1787826422.684956:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670563954048:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826422.694921:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826422.695076:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585274368 00000100:00100000:0.0:1787826422.695086:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585274368 00000100:00100000:0.0:1787826422.695144:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 510 00000100:00100000:0.0:1787826422.695155:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0161a40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:14864:x1874670585274368:12345-192.168.203.139@tcp:106: 00010000:00010000:0.0:1787826422.695171:0:42211:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826422.695185:0:42211:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00010000:0.0:1787826422.695306:0:42211:0:(cl_object.c:322:cl_object_glimpse()) header@00000000ca1bb7a3[0x110, 1, [0x200000405:0x38:0x0] hash] 00000020:00010000:0.0:1787826422.695309:0:42211:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826421 blocks: 0 00000020:00010000:0.0:1787826422.695313:0:42211:0:(cl_object.c:322:cl_object_glimpse()) header@0000000001146bd0[0x116, 2, [0x2c0000401:0x10c:0x0]] 00000020:00010000:0.0:1787826422.695314:0:42211:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826421 blocks: 0 00000100:00100000:0.0:1787826422.695374:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0161a40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:14864:x1874670585274368:12345-192.168.203.139@tcp:106: Request processed in 220us (461us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826422.695382:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 510 00010000:00010000:0.0:1787826422.700234:0:2401:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab1488 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.700257:0:2401:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab1488 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.700271:0:2401:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab1488 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.700279:0:2401:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab1488 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.700288:0:2401:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab1488 lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826422.700301:0:2401:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:0.0:1787826422.700308:0:2401:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1787826422.700369:0:2401:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab1488 lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.700378:0:2401:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab1488 lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.700387:0:2401:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab1488 lrc: 4/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.700394:0:2401:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925ce00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:0.0:1787826422.700400:0:2401:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab1488 lrc: 5/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.700418:0:2401:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787826422.700424:0:2401:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab1488 lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.700432:0:2401:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab1488 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826422.700462:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670563954048:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826422.705489:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826422.705725:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585274496 00000100:00100000:0.0:1787826422.705734:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585274496 00000100:00100000:0.0:1787826422.705973:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 511 00000100:00100000:0.0:1787826422.705984:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0160b40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6520:x1874670585274496:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826422.706005:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1473 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82f89 expref: -99 pid: 55664 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826422.706085:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826422.706091:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0160b40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6520:x1874670585274496:12345-192.168.203.139@tcp:104: Request processed in 110us (611us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826422.706099:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 511 00000100:00100000:3.0:1787826422.706376:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563954432:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826422.712712:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563954432:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826422.727795:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826422.728239:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585274624 00000100:00100000:0.0:1787826422.728249:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585274624 00000100:00100000:0.0:1787826422.728307:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 512 00000100:00100000:0.0:1787826422.728317:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0163480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6518:x1874670585274624:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826422.728336:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab1496 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82fe4 expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826422.728449:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826422.728456:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0163480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6518:x1874670585274624:12345-192.168.203.139@tcp:104: Request processed in 141us (671us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826422.728464:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 512 00010000:00010000:3.0:1787826422.728789:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826422.728797:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab1496 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82fe4 expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.728806:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515800 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826422.728812:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab1496 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x3a:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82fe4 expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.728820:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab1496 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x3a:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82fe4 expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.728953:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab1496 lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x3a:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b82fe4 expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826422.728978:0:42220:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x3a:0x0]: losing layout lock 00010000:00010000:0.0:1787826422.729095:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab1496 lrc: 2/0,0 mode: --/PR res: [0x200000405:0x3a:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82fe4 expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826422.729103:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1787826422.729161:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563955072:192.168.203.139@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1787826422.729274:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab1496 lrc: 1/0,0 mode: --/PR res: [0x200000405:0x3a:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82fe4 expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826422.729284:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab1496 lrc: 0/0,0 mode: --/PR res: [0x200000405:0x3a:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82fe4 expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826422.729298:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826422.729300:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826422.729302:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826422.733389:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563955072:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826422.750906:0:55670:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa1c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55670:1874670563955328:192.168.203.139@tcp:35:lfs.0 00000002:00100000:3.0:1787826422.750944:0:55670:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889ef9a40 x1874670563954944/t8589935138(8589935138) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826438 ref 2 fl Complete:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1787826422.750982:0:55670:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826422.750991:0:55670:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:3.0:1787826422.751007:0:55670:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826422.751020:0:55670:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 30 res 8589935621/29 (0 0) 00010000:00010000:3.0:1787826422.751067:0:55670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab14a4 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 55670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.751076:0:55670:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab14a4 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.751086:0:55670:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab14a4 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55670 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826422.751105:0:55670:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efda40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55670:1874670563955456:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826422.751173:0:55670:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25900 going to sleep for 11 seconds 00000100:00100000:0.0:1787826422.758887:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826422.758976:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585274752 00000100:00100000:1.0:1787826422.758986:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585274752 00000100:00100000:1.0:1787826422.759045:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 513 00000100:00100000:1.0:1787826422.759056:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aeed00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:8247:x1874670585274752:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826422.759076:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab148f lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x22153cd396b82fcf expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826422.760278:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826422.760292:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aeed00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:8247:x1874670585274752:12345-192.168.203.139@tcp:104: Request processed in 1235us (1413us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826422.760302:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 513 00010000:00010000:3.0:1787826422.761637:0:49796:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826422.761650:0:49796:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab148f lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x22153cd396b82fcf expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.761661:0:49796:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514200 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826422.761666:0:49796:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab148f lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x22153cd396b82fcf expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.761680:0:49796:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab148f lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x22153cd396b82fcf expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.761690:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab148f lrc: 3/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x22153cd396b82fcf expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787826422.762255:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef83c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563955584:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826422.771291:0:55670:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efda40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55670:1874670563955456:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826422.771325:0:55670:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab14a4 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8300e expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.771340:0:55670:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab14a4 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8300e expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.771357:0:55670:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab14a4 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8300e expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.771368:0:55670:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab14a4 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8300e expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.771385:0:55670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab14a4 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8300e expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.771394:0:55670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab14a4 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8300e expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.771411:0:55670:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab14a4 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8300e expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826422.771426:0:55670:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff976889efda40 x1874670563955456/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 576/720 e 0 to 0 dl 1787826438 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826422.771445:0:55670:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab14a4 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8300e expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826422.771458:0:55670:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab14a4 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8300e expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.771475:0:55670:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826422.771482:0:55670:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826422.771537:0:55670:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 29 -> 29 00000080:00010000:3.0:1787826422.771542:0:55670:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab14a4 00000080:00010000:3.0:1787826422.771550:0:55670:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:3.0:1787826422.771553:0:55670:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab14a4 from it ffffb2d4421c3cd8 00010000:00010000:3.0:1787826422.771557:0:55670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab14a4 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b8300e expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.771566:0:55670:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab14a4 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8300e expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826422.771615:0:55670:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1787826422.771624:0:55670:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:3.0:1787826422.771645:0:55670:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1787826422.771648:0:55670:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff9768837ff2b0, osc lock ffff9768984eeea0, flags 0x400001002 00010000:00010000:3.0:1787826422.771665:0:55670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826422.771679:0:55670:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826422.771689:0:55670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826422.771697:0:55670:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826422.771740:0:55670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006be8f0c7/0x8e7c99ee62ab14ab lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.771752:0:55670:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006be8f0c7/0x8e7c99ee62ab14ab lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787826422.771779:0:55670:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1787826422.778855:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826422.779085:0:42211:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:0.0:1787826422.779106:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585274880 00000100:00100000:0.0:1787826422.779115:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585274880 00000100:00100000:0.0:1787826422.779217:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 514 00000100:00100000:0.0:1787826422.779228:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff97688a33f850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:8437:x1874670585274880:12345-192.168.203.139@tcp:106: 00010000:00010000:0.0:1787826422.779244:0:42211:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826422.779258:0:42211:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00000020:00010000:0.0:1787826422.779286:0:42211:0:(cl_object.c:322:cl_object_glimpse()) header@00000000ca1bb7a3[0x110, 1, [0x200000405:0x38:0x0] hash] 00000020:00010000:0.0:1787826422.779288:0:42211:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826421 blocks: 0 00000020:00010000:0.0:1787826422.779292:0:42211:0:(cl_object.c:322:cl_object_glimpse()) header@0000000001146bd0[0x116, 2, [0x2c0000401:0x10c:0x0]] 00000020:00010000:0.0:1787826422.779294:0:42211:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826421 blocks: 0 00000100:00100000:0.0:1787826422.779355:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff97688a33f850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:8437:x1874670585274880:12345-192.168.203.139@tcp:106: Request processed in 129us (533us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826422.779363:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 514 00000080:00200000:3.0:1787826422.788155:0:55670:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:3.0:1787826422.788164:0:55670:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9768983a0000 (ffff976883b9d518/0) at cl_lock_release():245 00000020:00010000:3.0:1787826422.788169:0:55670:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768983a0000 (ffff976883b9d518/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826422.788175:0:55670:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768837ff2b0 (ffff976883b9d998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1787826422.788179:0:55670:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768983a0000 (ffff976883b9d518/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826422.788182:0:55670:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768837ff2b0 (ffff976883b9d518/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826422.788195:0:55670:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 29 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1787826422.792563:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef83c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563955968:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826422.795555:0:2403:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab14b9 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.795582:0:2403:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab14b9 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8302a expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.795597:0:2403:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab14b9 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8302a expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.795608:0:2403:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab14b9 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8302a expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826422.795622:0:2403:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab14b9 lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b8302a expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787826422.795634:0:2403:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826422.795703:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab14b9 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b8302a expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.795722:0:2403:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab14b9 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b8302a expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.795730:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab14b9 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b8302a expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.795744:0:2403:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab14b9 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b8302a expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787826422.795765:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef83c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563955968:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826422.811742:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563956352:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826422.816629:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826422.816732:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585275008 00000100:00100000:0.0:1787826422.816742:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585275008 00000100:00100000:0.0:1787826422.816796:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 515 00000100:00100000:0.0:1787826422.816806:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0162d00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:28853:x1874670585275008:12345-192.168.203.139@tcp:106: 00010000:00010000:0.0:1787826422.816822:0:53061:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826422.816869:0:53061:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00000020:00010000:0.0:1787826422.816902:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@00000000ca1bb7a3[0x110, 1, [0x200000405:0x38:0x0] hash] 00000020:00010000:0.0:1787826422.816904:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826421 blocks: 0 00000020:00010000:0.0:1787826422.816908:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@0000000001146bd0[0x116, 2, [0x2c0000401:0x10c:0x0]] 00000020:00010000:0.0:1787826422.816910:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826421 blocks: 0 00000100:00100000:0.0:1787826422.816969:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0162d00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:28853:x1874670585275008:12345-192.168.203.139@tcp:106: Request processed in 164us (348us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826422.816976:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 515 00010000:00010000:3.0:1787826422.822112:0:2404:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab14c7 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.822142:0:2404:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab14c7 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.822155:0:2404:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab14c7 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.822162:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab14c7 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.822169:0:2404:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab14c7 lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826422.822181:0:2404:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:3.0:1787826422.822188:0:2404:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826422.822268:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab14c7 lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.822277:0:2404:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab14c7 lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.822285:0:2404:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab14c7 lrc: 4/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.822292:0:2404:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515000 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826422.822297:0:2404:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab14c7 lrc: 5/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.822310:0:2404:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826422.822317:0:2404:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab14c7 lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.822326:0:2404:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab14c7 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826422.822348:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563956352:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826422.824483:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563956480:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826422.829575:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826422.829670:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585275136 00000100:00100000:0.0:1787826422.829679:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585275136 00000100:00100000:0.0:1787826422.829761:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 516 00000100:00100000:0.0:1787826422.829772:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0160780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:30694:x1874670585275136:12345-192.168.203.139@tcp:106: 00010000:00010000:0.0:1787826422.829788:0:42211:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826422.829801:0:42211:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00000020:00010000:0.0:1787826422.829890:0:42211:0:(cl_object.c:322:cl_object_glimpse()) header@00000000ca1bb7a3[0x110, 1, [0x200000405:0x38:0x0] hash] 00000020:00010000:0.0:1787826422.829893:0:42211:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826421 blocks: 0 00000020:00010000:0.0:1787826422.829896:0:42211:0:(cl_object.c:322:cl_object_glimpse()) header@0000000001146bd0[0x116, 2, [0x2c0000401:0x10c:0x0]] 00000020:00010000:0.0:1787826422.829898:0:42211:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826421 blocks: 0 00000100:00100000:0.0:1787826422.830113:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0160780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:30694:x1874670585275136:12345-192.168.203.139@tcp:106: Request processed in 342us (545us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826422.830129:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 516 00010000:00010000:3.0:1787826422.838476:0:2403:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab14ce lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.838499:0:2403:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab14ce lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.838514:0:2403:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab14ce lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.838522:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab14ce lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.838530:0:2403:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab14ce lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826422.838543:0:2403:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:3.0:1787826422.838551:0:2403:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826422.838569:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab14ce lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.838577:0:2403:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab14ce lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.838586:0:2403:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab14ce lrc: 4/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.838593:0:2403:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514000 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826422.838599:0:2403:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab14ce lrc: 5/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.838616:0:2403:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826422.838623:0:2403:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab14ce lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.838631:0:2403:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab14ce lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826422.838650:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563956480:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826422.839994:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563956608:192.168.203.139@tcp:1:lfs.0 00000100:00100000:1.0:1787826422.847127:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826422.847572:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585275264 00000100:00100000:0.0:1787826422.847584:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585275264 00000100:00100000:1.0:1787826422.847718:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 517 00000100:00100000:1.0:1787826422.847731:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0161680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:30695:x1874670585275264:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826422.847748:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab1426 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x22153cd396b82e7f expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1787826422.848068:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826422.848076:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0161680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:30695:x1874670585275264:12345-192.168.203.139@tcp:104: Request processed in 349us (965us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826422.848092:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 517 00000100:00100000:3.0:1787826422.848570:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787826422.848587:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563956736:192.168.203.139@tcp:103:ldlm_bl.0 00000080:00200000:0.0:1787826422.856347:0:55670:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:0.0:1787826422.856358:0:55670:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:0.0:1787826422.856500:0:55670:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826422.856509:0:55670:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:0.0:1787826422.856530:0:55670:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826422.856559:0:55670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab14a4 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8300e expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826422.856572:0:55670:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab14a4 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8300e expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826422.856580:0:55670:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826422.856589:0:55670:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:0.0:1787826422.856592:0:55670:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab14a4 00000080:00010000:0.0:1787826422.856599:0:55670:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab14a4 from it ffffb2d4421c3cd8 00010000:00010000:0.0:1787826422.856603:0:55670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab14a4 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b8300e expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826422.856611:0:55670:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab14a4 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8300e expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1787826422.856635:0:55670:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 29 restore needed 0 00000001:00010000:0.0:1787826422.856643:0:55670:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:0.0:1787826422.856659:0:55670:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826422.856662:0:55670:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e55730, osc lock ffff9768849a3e78, flags 0x400001002 00010000:00010000:0.0:1787826422.856702:0:55670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000008e0679c0/0x8e7c99ee62ab14d5 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826422.856710:0:55670:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000008e0679c0/0x8e7c99ee62ab14d5 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826422.856746:0:55670:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1787826422.856922:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1787826422.856938:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563956864:192.168.203.139@tcp:101:lfs.0 00010000:00010000:1.0:1787826422.862498:0:2402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000008e0679c0/0x8e7c99ee62ab14d5 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826422.862524:0:2402:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000008e0679c0/0x8e7c99ee62ab14d5 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8305b expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826422.862539:0:2402:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000008e0679c0/0x8e7c99ee62ab14d5 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8305b expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826422.862550:0:2402:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000008e0679c0/0x8e7c99ee62ab14d5 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8305b expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1787826422.862565:0:2402:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000008e0679c0/0x8e7c99ee62ab14d5 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b8305b expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1787826422.862577:0:2402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1787826422.862632:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000008e0679c0/0x8e7c99ee62ab14d5 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b8305b expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826422.862640:0:2402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000008e0679c0/0x8e7c99ee62ab14d5 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b8305b expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826422.862647:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000008e0679c0/0x8e7c99ee62ab14d5 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b8305b expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826422.862655:0:2402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000008e0679c0/0x8e7c99ee62ab14d5 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b8305b expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1787826422.862685:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563956864:192.168.203.139@tcp:101:lfs.0 00000080:00200000:0.0:1787826422.862750:0:55670:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787826422.862756:0:55670:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf2780 (ffff976890d10758/0) at cl_lock_release():245 00000020:00010000:0.0:1787826422.862760:0:55670:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf2780 (ffff976890d10758/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826422.862765:0:55670:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e55730 (ffff97688386c098/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826422.862769:0:55670:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf2780 (ffff976890d10758/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826422.862772:0:55670:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e55730 (ffff976890d10758/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826422.862785:0:55670:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 29 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:0.0:1787826422.924638:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826422.924777:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585275520 00000100:00100000:1.0:1787826422.924786:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585275520 00000100:00100000:1.0:1787826422.924869:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 518 00000100:00100000:1.0:1787826422.924879:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aeda40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:30398:x1874670585275520:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826422.924898:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab142d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000030000 nid: local remote: 0x22153cd396b82e8d expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826422.924966:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826422.924972:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aeda40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:30398:x1874670585275520:12345-192.168.203.139@tcp:104: Request processed in 95us (345us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826422.924979:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 518 00010000:00010000:3.0:1787826422.925105:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826422.925113:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab142d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000030000 nid: local remote: 0x22153cd396b82e8d expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.925122:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925da00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826422.925128:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab142d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400030000 nid: local remote: 0x22153cd396b82e8d expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.925175:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab142d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400030020 nid: local remote: 0x22153cd396b82e8d expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.925182:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826422.925213:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab142d lrc: 1/0,0 mode: --/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400030020 nid: local remote: 0x22153cd396b82e8d expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.925222:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab142d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400030020 nid: local remote: 0x22153cd396b82e8d expref: -99 pid: 55654 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.925242:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826422.925244:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826422.925245:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826422.925260:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563957120:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826422.931928:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563957120:192.168.203.139@tcp:103:ldlm_bl.0 00000080:00200000:0.0:1787826422.936290:0:55670:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x3a:0x0] updating i_size 0 i_blocks 0 00000080:00200000:0.0:1787826422.936305:0:55670:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x3a:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1787826422.949138:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563957376:192.168.203.139@tcp:1:lfs.0 00000100:00100000:3.0:1787826422.956266:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563957376:192.168.203.139@tcp:1:lfs.0 00010000:00010000:3.0:1787826422.978189:0:55670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab14dc lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x3a:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83077 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.978212:0:55670:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab14dc lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x3a:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83077 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1787826422.978222:0:55670:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x3a:0x0](000000003f9764e1) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab14dc lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x3a:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83077 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826422.978264:0:55670:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x3a:0x0]: layout version change: 4294967294 -> 1 00010000:00010000:3.0:1787826422.978270:0:55670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab14dc lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x3a:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83077 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826422.978284:0:55670:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab14dc lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x3a:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83077 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:3.0:1787826422.978306:0:55670:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826422.978322:0:55670:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:3.0:1787826422.978335:0:55670:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1787826422.978339:0:55670:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff9768837ff2b0, osc lock ffff9768984eeea0, flags 0x0 00010000:00010000:3.0:1787826422.978351:0:55670:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976889533000 type 11 mode 2 res 10737419265/268 (0 18446744073709551615) 00010000:00010000:3.0:1787826422.978387:0:55670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.978397:0:55670:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.978407:0:55670:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826422.978504:0:55670:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55670:1874670563957888:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826422.978805:0:55670:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25300 going to sleep for 11 seconds 00000100:00100000:0.0:1787826422.985865:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826422.986242:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585275648 00000100:00100000:1.0:1787826422.986252:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585275648 00000100:00100000:1.0:1787826422.986310:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 519 00000100:00100000:1.0:1787826422.986320:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aeed00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:14864:x1874670585275648:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826422.986340:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab14b9 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b8302a expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826422.986812:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826422.986821:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aeed00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:14864:x1874670585275648:12345-192.168.203.139@tcp:104: Request processed in 502us (964us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826422.986858:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 519 00000100:00100000:3.0:1787826422.991172:0:55670:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55670:1874670563957888:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826422.991217:0:55670:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 4/0,1 mode: --/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.991232:0:55670:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 4/0,1 mode: --/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.991245:0:55670:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 4/0,1 mode: --/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826422.994007:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826422.994103:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585275776 00000100:00100000:1.0:1787826422.994112:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585275776 00000100:00100000:1.0:1787826422.994161:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 520 00000100:00100000:1.0:1787826422.994171:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aecf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:6504:x1874670585275776:12345-192.168.203.139@tcp:105: 00010000:00010000:1.0:1787826422.994189:0:42210:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 5/0,1 mode: --/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826422.994213:0:42210:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 5/0,1 mode: --/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826422.994224:0:42210:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 5/0,1 mode: --/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826422.994235:0:42210:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826422.994246:0:42210:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 6/0,1 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826422.994256:0:42210:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 6/0,1 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826422.994281:0:42210:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff976888fb9a00) 00000100:00100000:1.0:1787826422.994337:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aecf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:6504:x1874670585275776:12345-192.168.203.139@tcp:105: Request processed in 167us (338us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826422.994344:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 520 00010000:00010000:3.0:1787826422.994855:0:55670:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.994868:0:55670:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.994885:0:55670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826422.994897:0:55670:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.994910:0:55670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826422.994918:0:55670:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826422.995001:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826422.995028:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563958144:192.168.203.139@tcp:2:lfs.0 00000100:00100000:3.0:1787826423.000388:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563958144:192.168.203.139@tcp:2:lfs.0 00000100:00100000:3.0:1787826423.001343:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563958016:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826423.029119:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826423.036588:0:53061:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1787826423.036616:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585275904 00000100:00100000:1.0:1787826423.036626:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585275904 00000100:00100000:1.0:1787826423.036693:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 521 00000100:00100000:1.0:1787826423.036704:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768895b8850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8435:x1874670585275904:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826423.036721:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000008e0679c0/0x8e7c99ee62ab14d5 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b8305b expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787826423.036813:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826423.036820:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768895b8850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8435:x1874670585275904:12345-192.168.203.139@tcp:104: Request processed in 119us (7711us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826423.036853:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 521 00010000:00010000:3.0:1787826423.038024:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000008e0679c0/0x8e7c99ee62ab14d5 lrc: 2/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b8305b expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826423.038033:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826423.038065:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000008e0679c0/0x8e7c99ee62ab14d5 lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b8305b expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826423.038073:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000008e0679c0/0x8e7c99ee62ab14d5 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b8305b expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826423.038089:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826423.038091:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826423.038093:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826423.038100:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787826423.038116:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563958656:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826423.044447:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826423.044513:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585276032 00000100:00100000:1.0:1787826423.044520:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585276032 00000100:00100000:1.0:1787826423.044574:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 522 00000100:00100000:1.0:1787826423.044583:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aec000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:6505:x1874670585276032:12345-192.168.203.139@tcp:105: 00010000:00010000:1.0:1787826423.044679:0:42211:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab14ea lrc: 5/0,1 mode: --/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8308c expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.044690:0:42211:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab14ea lrc: 5/0,1 mode: --/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8308c expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.044700:0:42211:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab14ea lrc: 5/0,1 mode: --/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b8308c expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.044710:0:42211:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab14ea lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x22153cd396b8308c expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.044719:0:42211:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab14ea lrc: 6/0,1 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x22153cd396b8308c expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.044734:0:42211:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab14ea lrc: 6/0,1 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x22153cd396b8308c expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.044764:0:42211:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff976883515400) 00000100:00100000:1.0:1787826423.044860:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aec000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:6505:x1874670585276032:12345-192.168.203.139@tcp:105: Request processed in 277us (414us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826423.044868:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 522 00000100:00100000:3.0:1787826423.052348:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563958656:192.168.203.139@tcp:103:ldlm_bl.0 00000020:00010000:3.0:1787826423.052661:0:55670:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976890fb74e8 (ffff976890d10128/0) at cl_lock_release():245 00000020:00010000:3.0:1787826423.052666:0:55670:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976890fb74e8 (ffff976890d10128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826423.052672:0:55670:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757c8b0 (ffff976883b9d518/1) at cl_lock_cancel():142 00010000:00010000:3.0:1787826423.052682:0:55670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab14ea lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8308c expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.052703:0:55670:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab14ea lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8308c expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:3.0:1787826423.052719:0:55670:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976890fb74e8 (ffff976890d10128/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826423.052722:0:55670:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757c8b0 (ffff976890d10128/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826423.052738:0:55670:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 29 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1787826423.052753:0:55670:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8) current size 0, valid attrs 100101f0, mode 9768, uid 0, gid 0, new size 0, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826423.000000000, rc 0 00000080:00200080:3.0:1787826423.052780:0:55670:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=132a570 00010000:00010000:3.0:1787826423.052981:0:55670:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab14c0 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0x22153cd396b83038 expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.053107:0:55670:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab14c0 lrc: 0/0,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x85809400000000 nid: local remote: 0x22153cd396b83038 expref: -99 pid: 55670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826423.053129:0:55670:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889ef83c0 x1874670563956224/t8589935140(8589935140) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826438 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826423.053141:0:55670:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1787826423.053147:0:55670:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889eff0c0 x1874670563959040/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 504/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826423.053171:0:55670:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55670:1874670563959040:192.168.203.139@tcp:35:lfs.0 00000100:00100000:3.0:1787826423.053318:0:55670:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25300 going to sleep for 11 seconds 00000100:00100000:0.0:1787826423.066098:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826423.066157:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826423.066208:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585276160 00000100:00100000:0.0:1787826423.066218:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585276160 00000100:00100000:0.0:1787826423.066261:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585276288 00000100:00100000:0.0:1787826423.066265:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585276288 00000100:00100000:0.0:1787826423.066279:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 523 00000100:00100000:0.0:1787826423.066289:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aee940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+6:14257:x1874670585276160:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826423.066307:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1481 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82fb3 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826423.066387:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826423.066394:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aee940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+6:14257:x1874670585276160:12345-192.168.203.139@tcp:104: Request processed in 107us (307us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826423.066401:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 523 00000100:00100000:0.0:1787826423.066420:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 524 00000100:00100000:0.0:1787826423.066426:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aef480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585276288:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826423.066434:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab146c lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82f66 expref: -99 pid: 55662 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826423.066463:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:0.0:1787826423.066469:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aef480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585276288:12345-192.168.203.139@tcp:104: Request processed in 43us (313us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826423.066475:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 524 00010000:00010000:3.0:1787826423.066610:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1787826423.066618:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1481 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82fb3 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826423.066627:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925d600 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826423.066632:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1481 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82fb3 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826423.066641:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1481 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b82fb3 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826423.066650:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1481 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 3 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b82fb3 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826423.066675:0:42220:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab146c lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b82f66 expref: -99 pid: 55662 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000002:00100000:3.0:1787826423.066721:0:42220:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976882aec780 x1874670563953920/t8589935135(8589935135) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826438 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826423.066733:0:42220:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 2 is allocated opc 35, max 7 00000100:00100000:3.0:1787826423.066739:0:42220:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889effc00 x1874670563959296/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826423.066763:0:42220:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:42220:1874670563959296:192.168.203.139@tcp:35:ldlm_bl.0 00000100:00100000:3.0:1787826423.066812:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25540 going to sleep for 11 seconds 00000100:00100000:3.0:1787826423.073759:0:42220:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:42220:1874670563959296:192.168.203.139@tcp:35:ldlm_bl.0 00000002:00100000:3.0:1787826423.073794:0:42220:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976882aec780 x1874670563953920/t8589935135(8589935135) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826438 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1787826423.073898:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1481 lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82fb3 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826423.073917:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826423.074368:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1481 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82fb3 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826423.074378:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1481 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b82fb3 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826423.074398:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826423.074401:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826423.074404:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826423.080194:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826423.081022:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585276416 00000100:00100000:1.0:1787826423.081033:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585276416 00000100:00100000:1.0:1787826423.081092:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 525 00000100:00100000:1.0:1787826423.081102:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aef840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585276416:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826423.081121:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab14dc lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x3a:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83077 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826423.081207:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826423.081213:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aef840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585276416:12345-192.168.203.139@tcp:104: Request processed in 113us (1031us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826423.081221:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 525 00010000:00010000:3.0:1787826423.081772:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826423.081782:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab14dc lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x3a:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83077 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826423.081792:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514200 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826423.081797:0:42221:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab14dc lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x3a:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83077 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826423.081805:0:42221:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab14dc lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x3a:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83077 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826423.081814:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab14dc lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x3a:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b83077 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826423.081870:0:42221:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x3a:0x0]: losing layout lock 00010000:00010000:3.0:1787826423.081910:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab14dc lrc: 2/0,0 mode: --/CR res: [0x200000405:0x3a:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83077 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826423.081923:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826423.081942:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab14dc lrc: 1/0,0 mode: --/CR res: [0x200000405:0x3a:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83077 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826423.081950:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab14dc lrc: 0/0,0 mode: --/CR res: [0x200000405:0x3a:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83077 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826423.081972:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826423.081974:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826423.081976:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826423.081992:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563959680:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826423.088928:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563959680:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826423.110248:0:55670:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55670:1874670563959040:192.168.203.139@tcp:35:lfs.0 00000002:00100000:3.0:1787826423.110287:0:55670:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889ef83c0 x1874670563956224/t8589935140(8589935140) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826438 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1787826423.110328:0:55670:0:(file.c:7076:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000002:00010000:3.0:1787826423.110339:0:55670:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: layout flags 0 00010000:00010000:3.0:1787826423.110400:0:55670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.110411:0:55670:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.110549:0:55670:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55670 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826423.110573:0:55670:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55670:1874670563959808:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826423.110652:0:55670:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25300 going to sleep for 11 seconds 00000100:00100000:3.0:1787826423.114906:0:55670:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55670:1874670563959808:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826423.114940:0:55670:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b830a1 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.114955:0:55670:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830a1 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826423.114970:0:55670:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff976889eff0c0 x1874670563959808/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 376/584 e 0 to 0 dl 1787826439 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826423.114980:0:55670:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830a1 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826423.114992:0:55670:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830a1 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.115005:0:55670:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826423.115011:0:55670:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1787826423.115028:0:55670:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab14f1 00000080:00010000:3.0:1787826423.115037:0:55670:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab14f1 from it ffffb2d4421c3bc8 00010000:00010000:3.0:1787826423.115041:0:55670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b830a1 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.115050:0:55670:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830a1 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826423.115077:0:55670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830a1 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826423.115088:0:55670:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830a1 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1787826423.115096:0:55670:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830a1 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826423.115134:0:55670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b830a1 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826423.115142:0:55670:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830a1 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826423.115157:0:55670:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 29 restore needed 0 00020000:00200000:3.0:1787826423.115168:0:55670:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826423.115195:0:55670:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:3.0:1787826423.115223:0:55670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830a1 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826423.115232:0:55670:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830a1 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:3.0:1787826423.115240:0:55670:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830a1 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826423.115263:0:55670:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826423.115268:0:55670:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826423.115301:0:55670:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000008:00010000:3.0:1787826423.115315:0:55670:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 0 [cma]time: 1787826421 1787826412 1787826412 by oinfo size 0 blocks 0 [cma]time 1787826423 1787826412 1787826412 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab14ea lrc: 1/0,0 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8308c expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826423.115985:0:55670:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 4294967294 -> 30 00010000:00010000:3.0:1787826423.115994:0:55670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b830a1 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826423.116003:0:55670:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b830a1 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:3.0:1787826423.116036:0:55670:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 1154002 00000002:00100000:3.0:1787826423.116085:0:55670:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889ef9e00 x1874670563954304/t8589935136(8589935136) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826438 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826423.116097:0:55670:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1787826423.116103:0:55670:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889eff0c0 x1874670563960064/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826423.116125:0:55670:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55670:1874670563960064:192.168.203.139@tcp:35:lfs.0 00000100:00100000:3.0:1787826423.116223:0:55670:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25300 going to sleep for 11 seconds 00000100:00100000:3.0:1787826423.120855:0:55670:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55670:1874670563960064:192.168.203.139@tcp:35:lfs.0 00000002:00100000:3.0:1787826423.120891:0:55670:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889ef9e00 x1874670563954304/t8589935136(8589935136) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826438 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1787826423.120959:0:55670:0:(file.c:479:ll_file_release()) START file . :VOLATILE:0000:2D743009:fd=03:[0x200000405:0x3a:0x0](ffff976890b3a1c8), flags 554001 00000002:00100000:3.0:1787826423.121004:0:55670:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889ef8780 x1874670563954688/t8589935137(8589935137) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826438 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826423.121014:0:55670:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1787826423.121020:0:55670:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889eff0c0 x1874670563960320/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826423.121040:0:55670:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55670:1874670563960320:192.168.203.139@tcp:35:lfs.0 00000100:00100000:3.0:1787826423.121118:0:55670:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25300 going to sleep for 11 seconds 00000100:00100000:3.0:1787826423.134408:0:55670:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55670:1874670563960320:192.168.203.139@tcp:35:lfs.0 00000002:00100000:3.0:1787826423.134456:0:55670:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889ef8780 x1874670563954688/t8589935137(8589935137) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826438 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1787826423.183637:0:55672:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826423.183666:0:55672:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826423.183686:0:55672:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 243 lru_type: 1 00010000:00010000:1.0:1787826423.183701:0:55672:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 243 lru_type: 1 00000080:00200000:1.0:1787826423.183714:0:55672:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826423.183721:0:55672:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826423.183724:0:55672:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826423.183731:0:55672:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4421dbb58 00010000:00010000:1.0:1787826423.183735:0:55672:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 243 lru_type: 1 00010000:00010000:1.0:1787826423.183744:0:55672:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 244 lru_type: 1 00000080:00200000:1.0:1787826423.183758:0:55672:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826423.183765:0:55672:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=24 00000080:00200000:1.0:1787826423.183771:0:55672:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826423.183780:0:55672:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), flags=8 00000080:00200000:1.0:1787826423.183783:0:55672:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=getattr 00000002:00010000:1.0:1787826423.183798:0:55672:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f60b.sanity-flr,[0x0:0x0:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787826423.183868:0:55672:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55672 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.183877:0:55672:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55672 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.183888:0:55672:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55672 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826423.183914:0:55672:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0162d00 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55672:1874670563960448:192.168.203.139@tcp:101:getfattr.0 00000100:00100000:0.0:1787826423.184965:0:55672:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3a80 going to sleep for 11 seconds 00000100:00100000:1.0:1787826423.190787:0:55672:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0162d00 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55672:1874670563960448:192.168.203.139@tcp:101:getfattr.0 00010000:00010000:1.0:1787826423.190812:0:55672:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 4/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.190853:0:55672:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.190862:0:55672:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.190874:0:55672:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.190884:0:55672:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.190897:0:55672:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.190905:0:55672:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.190912:0:55672:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787826423.190923:0:55672:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9768b0162d00 x1874670563960448/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 584/824 e 0 to 0 dl 1787826439 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'getfattr.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787826423.190934:0:55672:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787826423.190959:0:55672:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.190969:0:55672:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/29 (0 0) 00000080:00200000:1.0:1787826423.190975:0:55672:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787826423.191020:0:55672:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x1d:0x0], updating i_size 0 00000080:00200000:1.0:1787826423.191024:0:55672:0:(namei.c:147:ll_iget()) got inode: [0x200000405:0x1d:0x0](ffff9768901a80c8): rc = 0 00000080:00200000:1.0:1787826423.191037:0:55672:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 30 -> 30 00000080:00010000:1.0:1787826423.191041:0:55672:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab14f8 00000080:00010000:1.0:1787826423.191051:0:55672:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:1.0:1787826423.191055:0:55672:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab14f8 from it ffffb2d4421dbb20 00010000:00010000:1.0:1787826423.191058:0:55672:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.191067:0:55672:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1787826423.191103:0:55672:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1787826423.191110:0:55672:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:1.0:1787826423.191131:0:55672:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826423.191141:0:55672:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826423.191151:0:55672:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787826423.191161:0:55672:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826423.191166:0:55672:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:1.0:1787826423.191169:0:55672:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab14f8 00000080:00010000:1.0:1787826423.191172:0:55672:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab14f8 from it ffffb2d4421dbcb8 00010000:00010000:1.0:1787826423.191175:0:55672:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826423.191182:0:55672:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787826423.191226:0:55672:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826423.191236:0:55672:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826423.191243:0:55672:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 244 lru_type: 1 00010000:00010000:1.0:1787826423.191251:0:55672:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 244 lru_type: 1 00000080:00200000:1.0:1787826423.191257:0:55672:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826423.191261:0:55672:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826423.191264:0:55672:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826423.191268:0:55672:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4421dbbf8 00010000:00010000:1.0:1787826423.191271:0:55672:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 244 lru_type: 1 00010000:00010000:1.0:1787826423.191278:0:55672:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 245 lru_type: 1 00000080:00200000:1.0:1787826423.191287:0:55672:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826423.191293:0:55672:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:1.0:1787826423.191297:0:55672:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826423.191299:0:55672:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=9 00000080:00200000:1.0:1787826423.191306:0:55672:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:1.0:1787826423.191312:0:55672:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200000:1.0:1787826423.191328:0:55672:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826423.191335:0:55672:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826423.191342:0:55672:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 245 lru_type: 1 00010000:00010000:1.0:1787826423.191350:0:55672:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 245 lru_type: 1 00000080:00200000:1.0:1787826423.191356:0:55672:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826423.191359:0:55672:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826423.191362:0:55672:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826423.191365:0:55672:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4421dbbf8 00010000:00010000:1.0:1787826423.191368:0:55672:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 245 lru_type: 1 00010000:00010000:1.0:1787826423.191375:0:55672:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 246 lru_type: 1 00000080:00200000:1.0:1787826423.191394:0:55672:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826423.191399:0:55672:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:1.0:1787826423.191402:0:55672:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826423.191405:0:55672:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=9 00000080:00200000:1.0:1787826423.191412:0:55672:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:1.0:1787826423.191415:0:55672:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200000:0.0:1787826423.258147:0:55678:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826423.258182:0:55678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826423.258212:0:55678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 248 lru_type: 1 00010000:00010000:0.0:1787826423.258231:0:55678:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 248 lru_type: 1 00000080:00200000:0.0:1787826423.258239:0:55678:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826423.258247:0:55678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826423.258250:0:55678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826423.258257:0:55678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4421c3bf8 00010000:00010000:0.0:1787826423.258261:0:55678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 248 lru_type: 1 00010000:00010000:0.0:1787826423.258269:0:55678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 249 lru_type: 1 00000080:00200000:0.0:1787826423.258285:0:55678:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826423.258292:0:55678:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:0.0:1787826423.258298:0:55678:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826423.258302:0:55678:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:0.0:1787826423.258306:0:55678:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 44 00000080:00200008:0.0:1787826423.258311:0:55678:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154400 00000002:00010000:0.0:1787826423.258326:0:55678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114401 00010000:00010000:0.0:1787826423.258379:0:55678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab14ff lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 55678 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.258387:0:55678:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab14ff lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55678 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826423.258400:0:55678:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826423.258413:0:55678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aed680 x1874670563960832/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826423.258423:0:55678:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab14ff lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55678 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826423.258444:0:55678:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563960832:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826423.258622:0:55678:0:(client.c:2715:ptlrpc_set_wait()) set ffff976891c9acc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826423.264587:0:55678:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563960832:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826423.264613:0:55678:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab14ff lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830e7 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.264628:0:55678:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab14ff lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b830e7 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.264637:0:55678:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab14ff lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830e7 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826423.264652:0:55678:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff976882aed680 x1874670563960832/t8589935149(8589935149) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826423.264673:0:55678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aed680 x1874670563960832/t8589935149(8589935149) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826423.264685:0:55678:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab14ff lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830e7 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826423.264699:0:55678:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab14ff lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830e7 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.264712:0:55678:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826423.264719:0:55678:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826423.264763:0:55678:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x1d:0x0], updating i_size 0 00000080:00200000:3.0:1787826423.264777:0:55678:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 30 -> 30 00000080:00010000:3.0:1787826423.264783:0:55678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab14ff 00000080:00010000:3.0:1787826423.264790:0:55678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab14ff from it ffffb2d4421c3bf0 00010000:00010000:3.0:1787826423.264794:0:55678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab14ff lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b830e7 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.264804:0:55678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab14ff lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b830e7 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000002:00100000:3.0:1787826423.264870:0:55678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aed680 x1874670563960832/t8589935149(8589935149) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1787826423.265024:0:55678:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:3.0:1787826423.265033:0:55678:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:3.0:1787826423.265068:0:55678:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826423.265075:0:55678:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:3.0:1787826423.265095:0:55678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826423.265107:0:55678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826423.265118:0:55678:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826423.265124:0:55678:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826423.265131:0:55678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:3.0:1787826423.265135:0:55678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab14f8 00000080:00010000:3.0:1787826423.265139:0:55678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab14f8 from it ffffb2d4421c3cd8 00010000:00010000:3.0:1787826423.265143:0:55678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826423.265150:0:55678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:3.0:1787826423.265194:0:55678:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154000 00010000:00010000:3.0:1787826423.265217:0:55678:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab14ff lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b830e7 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826423.265339:0:55678:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826423.265350:0:55678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826423.265359:0:55678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 249 lru_type: 1 00010000:00010000:3.0:1787826423.265368:0:55678:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 249 lru_type: 1 00000080:00200000:3.0:1787826423.265380:0:55678:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826423.265385:0:55678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826423.265388:0:55678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826423.265392:0:55678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4421c3bf8 00010000:00010000:3.0:1787826423.265395:0:55678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 249 lru_type: 1 00010000:00010000:3.0:1787826423.265402:0:55678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 250 lru_type: 1 00000080:00200000:3.0:1787826423.265410:0:55678:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826423.265416:0:55678:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:3.0:1787826423.265420:0:55678:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787826423.265423:0:55678:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:3.0:1787826423.265427:0:55678:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 44 00000080:00200008:3.0:1787826423.265433:0:55678:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154400 00000080:00200000:3.0:1787826423.265460:0:55678:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:3.0:1787826423.265464:0:55678:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:3.0:1787826423.265478:0:55678:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826423.265482:0:55678:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:3.0:1787826423.265491:0:55678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826423.265498:0:55678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826423.265505:0:55678:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826423.265517:0:55678:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826423.265521:0:55678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:3.0:1787826423.265524:0:55678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab14f8 00000080:00010000:3.0:1787826423.265528:0:55678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab14f8 from it ffffb2d4421c3cd8 00010000:00010000:3.0:1787826423.265531:0:55678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826423.265539:0:55678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200008:3.0:1787826423.265559:0:55678:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154000 00010000:00010000:3.0:1787826423.265568:0:55678:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab14ff lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b830e7 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826423.265594:0:55678:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826423.265601:0:55678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826423.265608:0:55678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 250 lru_type: 1 00010000:00010000:3.0:1787826423.265616:0:55678:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 250 lru_type: 1 00000080:00200000:3.0:1787826423.265622:0:55678:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826423.265626:0:55678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826423.265629:0:55678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826423.265633:0:55678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4421c3bf8 00010000:00010000:3.0:1787826423.265637:0:55678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 250 lru_type: 1 00010000:00010000:3.0:1787826423.265644:0:55678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 251 lru_type: 1 00000080:00200000:3.0:1787826423.265652:0:55678:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826423.265656:0:55678:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:3.0:1787826423.265660:0:55678:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787826423.265663:0:55678:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:3.0:1787826423.265666:0:55678:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 46 00000080:00200008:3.0:1787826423.265670:0:55678:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154402 00000002:00010000:3.0:1787826423.265681:0:55678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114403 00010000:00010000:3.0:1787826423.265725:0:55678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1506 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 55678 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.265732:0:55678:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1506 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55678 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787826423.265742:0:55678:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787826423.265751:0:55678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efd2c0 x1874670563961088/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787826423.265766:0:55678:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1506 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55678 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826423.265786:0:55678:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563961088:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826423.265967:0:55678:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25300 going to sleep for 11 seconds 00000100:00100000:0.0:1787826423.278876:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826423.278973:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585276928 00000100:00100000:1.0:1787826423.278984:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585276928 00000100:00100000:1.0:1787826423.279049:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 526 00000100:00100000:1.0:1787826423.279060:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aee1c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6518:x1874670585276928:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826423.279081:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787826423.279167:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826423.279173:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aee1c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6518:x1874670585276928:12345-192.168.203.139@tcp:104: Request processed in 116us (307us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826423.279182:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 526 00010000:00010000:3.0:1787826423.279359:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826423.279367:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826423.279377:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976890fde600 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826423.279391:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826423.279400:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826423.279409:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826423.279435:0:42220:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab14ff lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b830e7 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826423.279478:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 2/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826423.279486:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826423.279515:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 1/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826423.279523:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab14f8 lrc: 0/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b830c4 expref: -99 pid: 55672 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826423.279541:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826423.279543:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826423.279545:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826423.279554:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826423.279569:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563961216:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826423.283338:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563961216:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826423.284815:0:55678:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563961088:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826423.284870:0:55678:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode CW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1506 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830fc expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.284884:0:55678:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1506 lrc: 4/1,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830fc expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.284895:0:55678:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1506 lrc: 4/1,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b830fc expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.284904:0:55678:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1506 lrc: 4/1,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830fc expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.284916:0:55678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1506 lrc: 5/1,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b830fc expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.284924:0:55678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1506 lrc: 5/1,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b830fc expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.284930:0:55678:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1506 lrc: 4/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b830fc expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826423.284954:0:55678:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff976889efd2c0 x1874670563961088/t8589935150(8589935150) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826423.284974:0:55678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efd2c0 x1874670563961088/t8589935150(8589935150) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826423.284990:0:55678:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1506 lrc: 3/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830fc expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.285005:0:55678:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 8 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826423.285012:0:55678:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826423.285059:0:55678:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x1d:0x0], updating i_size 0 00000080:00010000:3.0:1787826423.285065:0:55678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1506 00000080:00010000:3.0:1787826423.285072:0:55678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1506 from it ffffb2d4421c3bf0 00010000:00010000:3.0:1787826423.285076:0:55678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1506 lrc: 3/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b830fc expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.285085:0:55678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1506 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b830fc expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:3.0:1787826423.285113:0:55678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efd2c0 x1874670563961088/t8589935150(8589935150) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:3.0:1787826423.285252:0:55678:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=800466af arg=7ffee598172c 00000080:00200080:3.0:1787826423.285269:0:55678:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=80086601 arg=7ffee59816c8 00000100:00100000:3.0:1787826423.285314:0:55678:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563961344:192.168.203.139@tcp:33:lfs.0 00000100:00100000:3.0:1787826423.285402:0:55678:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25300 going to sleep for 11 seconds 00000100:00100000:3.0:1787826423.287889:0:55678:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563961344:192.168.203.139@tcp:33:lfs.0 00000080:00200080:3.0:1787826423.287934:0:55678:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=7ffee59816b4 00000002:00010000:3.0:1787826423.287973:0:55678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 01004040000002 00010000:00010000:3.0:1787826423.287992:0:55678:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1506 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x22153cd396b830fc expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826423.288021:0:55678:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab14ff lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b830e7 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826423.288034:0:55678:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b830a1 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000002:00100000:3.0:1787826423.288072:0:55678:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976882aed680 x1874670563960832/t8589935149(8589935149) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826439 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826423.288083:0:55678:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1787826423.288088:0:55678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efe940 x1874670563961600/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826423.288108:0:55678:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563961600:192.168.203.139@tcp:35:lfs.0 00000100:00100000:3.0:1787826423.288174:0:55678:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25300 going to sleep for 11 seconds 00000100:00100000:3.0:1787826423.291665:0:55678:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563961600:192.168.203.139@tcp:35:lfs.0 00000002:00100000:3.0:1787826423.291703:0:55678:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976882aed680 x1874670563960832/t8589935149(8589935149) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826439 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1787826423.291787:0:55678:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1506 lrc: 1/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b830fc expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826423.291797:0:55678:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab14ff lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b830e7 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826423.291803:0:55678:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1787826423.291807:0:55678:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1506 lrc: 0/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b830fc expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826423.291821:0:55678:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab14ff lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b830e7 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826423.291956:0:55678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab150d lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 55678 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.291963:0:55678:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x81000000001000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab150d lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 55678 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787826423.291973:0:55678:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787826423.291980:0:55678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efe940 x1874670563961856/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 976/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787826423.291989:0:55678:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab150d lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x81000000000000 pid: 55678 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826423.292009:0:55678:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563961856:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826423.292171:0:55678:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25300 going to sleep for 11 seconds 00000100:00100000:3.0:1787826423.301178:0:55678:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563961856:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826423.301218:0:55678:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab150d lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83111 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.301232:0:55678:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab150d lrc: 4/1,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83111 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.301246:0:55678:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab150d lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83111 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.301260:0:55678:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab150d lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83111 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.301273:0:55678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab150d lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83111 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.301282:0:55678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab150d lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83111 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.301290:0:55678:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab150d lrc: 4/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83111 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826423.301302:0:55678:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x600002b, status=0 req@ffff976889efe940 x1874670563961856/t8589935152(8589935152) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 976/848 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826423.301317:0:55678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efe940 x1874670563961856/t8589935152(8589935152) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 976/848 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826423.301332:0:55678:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab150d lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83111 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.301344:0:55678:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 2 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826423.301350:0:55678:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000002:00100000:3.0:1787826423.301384:0:55678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efe940 x1874670563961856/t8589935152(8589935152) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 768/848 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1787826423.301394:0:55678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab150d 00000080:00010000:3.0:1787826423.301400:0:55678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab150d from it ffffb2d4421c3cc0 00010000:00010000:3.0:1787826423.301405:0:55678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab150d lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83111 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.301412:0:55678:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab150d lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83111 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200080:3.0:1787826423.301475:0:55678:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=9672e0 00010000:00010000:3.0:1787826423.301524:0:55678:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab150d lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0x22153cd396b83111 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.301547:0:55678:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab150d lrc: 0/0,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x85809400000000 nid: local remote: 0x22153cd396b83111 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826423.301563:0:55678:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889efe940 x1874670563961856/t8589935152(8589935152) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 768/848 e 0 to 0 dl 1787826439 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826423.301571:0:55678:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1787826423.301577:0:55678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889effc00 x1874670563962112/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 504/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826423.301596:0:55678:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563962112:192.168.203.139@tcp:35:lfs.0 00000100:00100000:3.0:1787826423.301729:0:55678:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25300 going to sleep for 11 seconds 00000100:00100000:1.0:1787826423.325678:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826423.326801:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585277056 00000100:00100000:0.0:1787826423.326812:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585277056 00000100:00100000:0.0:1787826423.326891:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 527 00000100:00100000:0.0:1787826423.326900:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0162d00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6522:x1874670585277056:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826423.326917:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b830a1 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826423.327001:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826423.327007:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0162d00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6522:x1874670585277056:12345-192.168.203.139@tcp:104: Request processed in 109us (1339us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826423.327015:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 527 00010000:00010000:3.0:1787826423.330568:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826423.330582:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b830a1 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826423.330596:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976888fb8200 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826423.330602:0:42221:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b830a1 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826423.330617:0:42221:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b830a1 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826423.330626:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b830a1 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1787826423.330650:0:42221:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x1d:0x0]: losing layout lock 00010000:00010000:3.0:1787826423.330703:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b830a1 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826423.330711:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826423.331863:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b830a1 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826423.331878:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab14f1 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b830a1 expref: -99 pid: 55670 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826423.332032:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826423.332036:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826423.332038:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826423.340508:0:55678:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563962112:192.168.203.139@tcp:35:lfs.0 00000080:00020000:3.0:1787826423.340527:0:55678:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9768877a4800: inode [0x200000405:0x1d:0x0] mdc close failed: rc = -22 00000002:00100000:3.0:1787826423.340562:0:55678:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889efe940 x1874670563961856/t8589935152(8589935152) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 768/848 e 0 to 0 dl 1787826439 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1787826423.340600:0:55678:0:(file.c:7076:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000002:00010000:3.0:1787826423.340611:0:55678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: layout flags 0 00010000:00010000:3.0:1787826423.340659:0:55678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1514 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55678 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.340675:0:55678:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1514 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55678 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.340684:0:55678:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1514 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55678 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826423.340703:0:55678:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563962368:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826423.340781:0:55678:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25300 going to sleep for 11 seconds 00000100:00100000:3.0:1787826423.344769:0:55678:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563962368:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826423.344799:0:55678:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1514 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83126 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.344815:0:55678:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1514 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83126 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826423.344862:0:55678:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff976889efe940 x1874670563962368/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 376/584 e 0 to 0 dl 1787826439 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826423.344873:0:55678:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1514 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83126 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826423.344886:0:55678:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1514 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83126 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.344900:0:55678:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826423.344906:0:55678:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1787826423.344921:0:55678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1514 00000080:00010000:3.0:1787826423.344931:0:55678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1514 from it ffffb2d4421c3bc8 00010000:00010000:3.0:1787826423.344935:0:55678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1514 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83126 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.344946:0:55678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1514 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83126 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826423.344972:0:55678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1514 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83126 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826423.344984:0:55678:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1514 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83126 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1787826423.344992:0:55678:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1514 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83126 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826423.345031:0:55678:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 4294967294 -> 30 00010000:00010000:3.0:1787826423.345036:0:55678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1514 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83126 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826423.345043:0:55678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1514 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83126 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826423.345253:0:55678:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826423.345271:0:55678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826423.345280:0:55678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 251 lru_type: 1 00010000:00010000:3.0:1787826423.345290:0:55678:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 251 lru_type: 1 00000080:00200000:3.0:1787826423.345297:0:55678:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826423.345303:0:55678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826423.345306:0:55678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826423.345311:0:55678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4421c3bf8 00010000:00010000:3.0:1787826423.345314:0:55678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 251 lru_type: 1 00010000:00010000:3.0:1787826423.345320:0:55678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 252 lru_type: 1 00000080:00200000:3.0:1787826423.345329:0:55678:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826423.345336:0:55678:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=24 00000080:00200000:3.0:1787826423.345341:0:55678:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787826423.345353:0:55678:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787826423.345358:0:55678:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=. :VOLATILE:0000:D1188F8:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), file ffff97688773dc80, open_flags 2f9c2, mode 8180 opened 0 00000080:00200000:3.0:1787826423.345364:0:55678:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=. :VOLATILE:0000:D1188F8:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=open|creat 00000002:00010000:3.0:1787826423.345383:0:55678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: . :VOLATILE:0000:D1188F8:fd=03,[0x200000405:0x3b:0x0]) in obj [0x200000bd0:0x4:0x0], intent: open|creat flags 0574703 00010000:00010000:3.0:1787826423.345429:0:55678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab151b lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55678 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.345435:0:55678:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab151b lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55678 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787826423.345446:0:55678:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787826423.345452:0:55678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efe940 x1874670563962624/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/66640 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787826423.345465:0:55678:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab151b lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55678 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826423.345483:0:55678:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563962624:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826423.345789:0:55678:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25300 going to sleep for 11 seconds 00000100:00100000:3.0:1787826423.356551:0:55678:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563962624:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826423.356576:0:55678:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab151b lrc: 4/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83134 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.356601:0:55678:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab151b lrc: 4/0,1 mode: --/PR res: [0x200000405:0x3b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83134 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.356610:0:55678:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab151b lrc: 4/0,1 mode: --/PR res: [0x200000405:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83134 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.356621:0:55678:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab151b lrc: 4/0,1 mode: PR/PR res: [0x200000405:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83134 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.356631:0:55678:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab151b lrc: 4/0,1 mode: PR/PR res: [0x200000405:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83134 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.356644:0:55678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab151b lrc: 5/1,1 mode: PR/PR res: [0x200000405:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83134 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.356651:0:55678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab151b lrc: 5/1,1 mode: PR/PR res: [0x200000405:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83134 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.356664:0:55678:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab151b lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83134 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826423.356677:0:55678:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff976889efe940 x1874670563962624/t8589935154(8589935154) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826423.356690:0:55678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efe940 x1874670563962624/t8589935154(8589935154) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826423.356701:0:55678:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab151b lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83134 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.356711:0:55678:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/59 (0 0) 00000080:00200000:3.0:1787826423.356717:0:55678:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd0:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1787826423.356729:0:55678:0:(namei.c:1628:ll_create_it()) VFS Op:name=. :VOLATILE:0000:D1188F8:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=open|creat 00000080:00200000:3.0:1787826423.356759:0:55678:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000405:0x3b:0x0](ffff97689011c2c8) 00000080:00200000:3.0:1787826423.356766:0:55678:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x3b:0x0], updating i_size 0 00000080:00200000:3.0:1787826423.356796:0:55678:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x3b:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787826423.356800:0:55678:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000405:0x3b:0x0](ffff97689011c2c8) 00000080:00010000:3.0:1787826423.356803:0:55678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x3b:0x0](ffff97689011c2c8) for lock 0x8e7c99ee62ab151b 00000080:00010000:3.0:1787826423.356811:0:55678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x3b:0x0](ffff97689011c2c8) 00000080:00200008:3.0:1787826423.356864:0:55678:0:(file.c:1090:ll_file_open()) START file . :VOLATILE:0000:D1188F8:fd=03:[0x200000405:0x3b:0x0](ffff97689011c2c8), flags 574702 00000002:00100000:3.0:1787826423.356884:0:55678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efe940 x1874670563962624/t8589935154(8589935154) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1787826423.356917:0:55678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab151b from it ffff9768869d97e0 00010000:00010000:3.0:1787826423.356921:0:55678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab151b lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83134 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.356931:0:55678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab151b lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83134 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826423.356950:0:55678:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x3b:0x0](ffff97689011c2c8), inode mode 8180 mask 40 00000080:00200080:3.0:1787826423.356972:0:55678:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=7ffee59816b4 00000002:00010000:3.0:1787826423.356986:0:55678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 01004040000002 00010000:00010000:3.0:1787826423.357027:0:55678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1522 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 55678 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.357034:0:55678:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x81000000001000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1522 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 55678 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787826423.357043:0:55678:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787826423.357049:0:55678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efda40 x1874670563962880/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787826423.357058:0:55678:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1522 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x81000000000000 pid: 55678 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826423.357074:0:55678:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efda40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563962880:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826423.357240:0:55678:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25300 going to sleep for 11 seconds 00000100:00100000:3.0:1787826423.368882:0:55678:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efda40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563962880:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826423.368909:0:55678:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1522 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83149 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.368922:0:55678:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1522 lrc: 4/1,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83149 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.368940:0:55678:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1522 lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83149 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.368951:0:55678:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1522 lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83149 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.368964:0:55678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1522 lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83149 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.368972:0:55678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1522 lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83149 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.368980:0:55678:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1522 lrc: 4/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83149 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826423.368992:0:55678:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x600002b, status=0 req@ffff976889efda40 x1874670563962880/t8589935155(8589935155) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826423.369006:0:55678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efda40 x1874670563962880/t8589935155(8589935155) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826423.369021:0:55678:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1522 lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83149 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.369035:0:55678:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 2 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826423.369040:0:55678:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000002:00100000:3.0:1787826423.369061:0:55678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efda40 x1874670563962880/t8589935155(8589935155) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1787826423.369069:0:55678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1522 00000080:00010000:3.0:1787826423.369074:0:55678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1522 from it ffffb2d4421c3cc0 00010000:00010000:3.0:1787826423.369078:0:55678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1522 lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83149 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.369085:0:55678:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1522 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83149 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200080:3.0:1787826423.369138:0:55678:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=967300 00010000:00010000:3.0:1787826423.369182:0:55678:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1522 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0x22153cd396b83149 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.369207:0:55678:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1522 lrc: 0/0,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x85809400000000 nid: local remote: 0x22153cd396b83149 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826423.369224:0:55678:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889efda40 x1874670563962880/t8589935155(8589935155) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826439 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826423.369234:0:55678:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1787826423.369239:0:55678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efc780 x1874670563963136/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 504/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826423.369259:0:55678:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563963136:192.168.203.139@tcp:35:lfs.0 00000100:00100000:3.0:1787826423.369343:0:55678:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25300 going to sleep for 11 seconds 00000100:00100000:0.0:1787826423.371803:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826423.372000:0:53061:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1787826423.372020:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585277184 00000100:00100000:1.0:1787826423.372028:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585277184 00000100:00100000:1.0:1787826423.372082:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 528 00000100:00100000:1.0:1787826423.372092:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff97688a33f450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6521:x1874670585277184:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826423.372108:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1514 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83126 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826423.372180:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826423.372187:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff97688a33f450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6521:x1874670585277184:12345-192.168.203.139@tcp:104: Request processed in 97us (393us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826423.372195:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 528 00000100:00100000:1.0:1787826423.378605:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826423.379978:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585277312 00000100:00100000:0.0:1787826423.379987:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585277312 00000100:00100000:0.0:1787826423.380045:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 529 00000100:00100000:0.0:1787826423.380055:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0162d00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6521:x1874670585277312:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826423.380073:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab151b lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83134 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826423.380153:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826423.380160:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0162d00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6521:x1874670585277312:12345-192.168.203.139@tcp:104: Request processed in 107us (1564us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826423.380168:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 529 00010000:00010000:3.0:1787826423.380336:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826423.380344:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab151b lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83134 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826423.380357:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976888fb8200 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826423.380362:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab151b lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x3b:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83134 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826423.380383:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab151b lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x3b:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83134 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826423.380392:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab151b lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x3b:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b83134 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826423.380415:0:42220:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x3b:0x0]: losing layout lock 00010000:00010000:3.0:1787826423.380454:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab151b lrc: 2/0,0 mode: --/PR res: [0x200000405:0x3b:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83134 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826423.380461:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826423.380488:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab151b lrc: 1/0,0 mode: --/PR res: [0x200000405:0x3b:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83134 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826423.380496:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab151b lrc: 0/0,0 mode: --/PR res: [0x200000405:0x3b:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83134 expref: -99 pid: 55678 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826423.380514:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826423.380516:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826423.380517:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826423.380524:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787826423.380550:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563963392:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826423.394006:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563963392:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826423.394008:0:55678:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563963136:192.168.203.139@tcp:35:lfs.0 00000002:00100000:1.0:1787826423.394049:0:55678:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889efda40 x1874670563962880/t8589935155(8589935155) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826439 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1787826423.395024:0:55678:0:(file.c:7076:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000002:00010000:1.0:1787826423.395038:0:55678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: layout flags 0 00010000:00010000:1.0:1787826423.395188:0:55678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf400/0x8e7c99ee62ab1529 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55678 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.395212:0:55678:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf400/0x8e7c99ee62ab1529 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55678 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.395223:0:55678:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf400/0x8e7c99ee62ab1529 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55678 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826423.395246:0:55678:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563963520:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826423.395315:0:55678:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3f00 going to sleep for 11 seconds 00000100:00100000:1.0:1787826423.400928:0:55678:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0160780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563963520:192.168.203.139@tcp:101:lfs.0 00010000:00010000:1.0:1787826423.400960:0:55678:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf400/0x8e7c99ee62ab1529 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83165 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.400975:0:55678:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf400/0x8e7c99ee62ab1529 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83165 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787826423.400990:0:55678:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9768b0160780 x1874670563963520/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 376/480 e 0 to 0 dl 1787826439 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787826423.401001:0:55678:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf400/0x8e7c99ee62ab1529 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83165 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787826423.401017:0:55678:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf400/0x8e7c99ee62ab1529 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83165 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.401029:0:55678:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:1.0:1787826423.401036:0:55678:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1787826423.401051:0:55678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1529 00000080:00010000:1.0:1787826423.401058:0:55678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1529 from it ffffb2d4421c3bc8 00010000:00010000:1.0:1787826423.401062:0:55678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf400/0x8e7c99ee62ab1529 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83165 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.401072:0:55678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf400/0x8e7c99ee62ab1529 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83165 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826423.401095:0:55678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf400/0x8e7c99ee62ab1529 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83165 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826423.401105:0:55678:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf400/0x8e7c99ee62ab1529 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83165 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1787826423.401114:0:55678:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf400/0x8e7c99ee62ab1529 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83165 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826423.401148:0:55678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf400/0x8e7c99ee62ab1529 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83165 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826423.401155:0:55678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf400/0x8e7c99ee62ab1529 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83165 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787826423.401170:0:55678:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1787826423.401191:0:55678:0:(lov_io.c:693:lov_io_mirror_init()) [0x200000405:0x1d:0x0]: flr state: 1, move mirror from 1 to 1, have retried: 0, mirror count: 2 00020000:00200000:1.0:1787826423.401195:0:55678:0:(lov_io.c:750:lov_io_mirror_init()) use delayed RPC state for this IO 00020000:00200000:1.0:1787826423.401207:0:55678:0:(lov_io.c:1253:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:1.0:1787826423.401232:0:55678:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:1.0:1787826423.401244:0:55678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf400/0x8e7c99ee62ab1529 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83165 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826423.401253:0:55678:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf400/0x8e7c99ee62ab1529 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83165 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1787826423.401261:0:55678:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf400/0x8e7c99ee62ab1529 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83165 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787826423.401274:0:55678:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1787826423.401278:0:55678:0:(lov_io.c:693:lov_io_mirror_init()) [0x200000405:0x1d:0x0]: flr state: 1, move mirror from 1 to 1, have retried: 0, mirror count: 2 00020000:00200000:1.0:1787826423.401281:0:55678:0:(lov_io.c:750:lov_io_mirror_init()) use delayed RPC state for this IO 00020000:00200000:1.0:1787826423.401283:0:55678:0:(lov_io.c:1253:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:1.0:1787826423.401289:0:55678:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1787826423.401345:0:55678:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 4294967294 -> 31 00010000:00010000:1.0:1787826423.401350:0:55678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf400/0x8e7c99ee62ab1529 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83165 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826423.401358:0:55678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf400/0x8e7c99ee62ab1529 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83165 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:1.0:1787826423.401387:0:55678:0:(file.c:479:ll_file_release()) START file . :VOLATILE:0000:D1188F8:fd=03:[0x200000405:0x3b:0x0](ffff97689011c2c8), flags 554002 00000002:00100000:1.0:1787826423.401425:0:55678:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889efe940 x1874670563962624/t8589935154(8589935154) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826439 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1787826423.401435:0:55678:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1787826423.401440:0:55678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b0160780 x1874670563963776/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1787826423.401462:0:55678:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563963776:192.168.203.139@tcp:35:lfs.0 00000100:00100000:1.0:1787826423.401544:0:55678:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3900 going to sleep for 11 seconds 00000100:00100000:1.0:1787826423.405436:0:55678:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0160780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563963776:192.168.203.139@tcp:35:lfs.0 00000002:00100000:1.0:1787826423.405472:0:55678:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889efe940 x1874670563962624/t8589935154(8589935154) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826439 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1787826423.405527:0:55678:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154002 00000002:00100000:1.0:1787826423.405577:0:55678:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889efd2c0 x1874670563961088/t8589935150(8589935150) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826439 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1787826423.405586:0:55678:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1787826423.405591:0:55678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b0161680 x1874670563964032/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1787826423.405610:0:55678:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0161680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563964032:192.168.203.139@tcp:35:lfs.0 00000100:00100000:1.0:1787826423.405690:0:55678:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3900 going to sleep for 11 seconds 00000100:00100000:1.0:1787826423.408910:0:55678:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0161680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55678:1874670563964032:192.168.203.139@tcp:35:lfs.0 00000002:00100000:1.0:1787826423.408945:0:55678:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889efd2c0 x1874670563961088/t8589935150(8589935150) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826439 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1787826423.446557:0:55680:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826423.446565:0:55680:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e554b0, osc lock ffff9768849a3e78, flags 0x400001002 00010000:00010000:0.0:1787826423.446590:0:55680:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826423.446607:0:55680:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826423.446620:0:55680:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826423.446630:0:55680:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826423.446675:0:55680:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000a0fb0a46/0x8e7c99ee62ab1537 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55680 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.446685:0:55680:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000a0fb0a46/0x8e7c99ee62ab1537 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55680 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826423.446727:0:55680:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1787826423.446942:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563964288:192.168.203.139@tcp:101:getfattr.0 00000100:00100000:0.0:1787826423.453133:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826423.453252:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585277568 00000100:00100000:1.0:1787826423.454095:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585277568 00000100:00100000:1.0:1787826423.454154:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 530 00000100:00100000:1.0:1787826423.454163:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aecb40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30694:x1874670585277568:12345-192.168.203.139@tcp:106: 00010000:00010000:1.0:1787826423.454178:0:42210:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826423.454191:0:42210:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:1.0:1787826423.454240:0:42210:0:(cl_object.c:322:cl_object_glimpse()) header@00000000349e195f[0x110, 1, [0x200000405:0x3a:0x0] hash] 00000020:00010000:1.0:1787826423.454243:0:42210:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826422 blocks: 0 00000020:00010000:1.0:1787826423.454247:0:42210:0:(cl_object.c:322:cl_object_glimpse()) header@000000001fe7a5ce[0x116, 2, [0x280000401:0x10c:0x0]] 00000020:00010000:1.0:1787826423.454249:0:42210:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826422 blocks: 0 00000100:00100000:1.0:1787826423.454360:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aecb40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30694:x1874670585277568:12345-192.168.203.139@tcp:106: Request processed in 197us (1236us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826423.454368:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 530 00010000:00010000:1.0:1787826423.458143:0:2402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000a0fb0a46/0x8e7c99ee62ab1537 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55680 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.458166:0:2402:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000a0fb0a46/0x8e7c99ee62ab1537 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55680 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.458179:0:2402:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000a0fb0a46/0x8e7c99ee62ab1537 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55680 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.458191:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000a0fb0a46/0x8e7c99ee62ab1537 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55680 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.458208:0:2402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000a0fb0a46/0x8e7c99ee62ab1537 lrc: 5/1,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55680 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1787826423.458222:0:2402:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:1.0:1787826423.458229:0:2402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1787826423.458246:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000a0fb0a46/0x8e7c99ee62ab1537 lrc: 4/1,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55680 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.458253:0:2402:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000a0fb0a46/0x8e7c99ee62ab1537 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55680 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.458262:0:2402:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000a0fb0a46/0x8e7c99ee62ab1537 lrc: 4/0,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55680 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.458269:0:2402:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925d200 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:1.0:1787826423.458275:0:2402:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000a0fb0a46/0x8e7c99ee62ab1537 lrc: 5/0,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55680 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.458288:0:2402:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1787826423.458295:0:2402:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000a0fb0a46/0x8e7c99ee62ab1537 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55680 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787826423.458308:0:55680:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00010000:00010000:1.0:1787826423.458309:0:2402:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000a0fb0a46/0x8e7c99ee62ab1537 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55680 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:0.0:1787826423.458319:0:55680:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf18b0 (ffff976890d10368/0) at cl_lock_release():245 00000020:00010000:0.0:1787826423.458325:0:55680:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf18b0 (ffff976890d10368/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826423.458331:0:55680:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e554b0 (ffff976890d10f38/1) at cl_lock_cancel():142 00000100:00100000:1.0:1787826423.458333:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563964288:192.168.203.139@tcp:101:getfattr.0 00000020:00010000:0.0:1787826423.458337:0:55680:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf18b0 (ffff976890d10368/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826423.458343:0:55680:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e554b0 (ffff976890d10368/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826423.458359:0:55680:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 31 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826423.458403:0:55680:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826423.458425:0:55680:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826423.458441:0:55680:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 253 lru_type: 1 00010000:00010000:0.0:1787826423.458454:0:55680:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 253 lru_type: 1 00000080:00200000:0.0:1787826423.458468:0:55680:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826423.458476:0:55680:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826423.458480:0:55680:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826423.458486:0:55680:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4421f3bf8 00010000:00010000:0.0:1787826423.458490:0:55680:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 253 lru_type: 1 00010000:00010000:0.0:1787826423.458499:0:55680:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826423.458514:0:55680:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826423.458521:0:55680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:0.0:1787826423.458527:0:55680:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826423.458531:0:55680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=9 00000080:00200000:0.0:1787826423.458538:0:55680:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:0.0:1787826423.458544:0:55680:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200000:0.0:1787826423.458562:0:55680:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826423.458570:0:55680:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826423.458577:0:55680:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826423.458586:0:55680:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826423.458593:0:55680:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826423.458597:0:55680:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826423.458600:0:55680:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826423.458604:0:55680:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4421f3bf8 00010000:00010000:0.0:1787826423.458607:0:55680:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826423.458615:0:55680:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826423.458624:0:55680:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826423.458628:0:55680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:0.0:1787826423.458631:0:55680:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826423.458634:0:55680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=9 00000080:00200000:0.0:1787826423.458641:0:55680:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:0.0:1787826423.458645:0:55680:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200000:0.0:1787826423.530772:0:55686:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826423.530804:0:55686:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826423.530844:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826423.530861:0:55686:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826423.530870:0:55686:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826423.530877:0:55686:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826423.530881:0:55686:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826423.530888:0:55686:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d442213bf8 00010000:00010000:0.0:1787826423.530892:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826423.530906:0:55686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826423.530923:0:55686:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826423.530934:0:55686:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:0.0:1787826423.530939:0:55686:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826423.530943:0:55686:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:0.0:1787826423.530947:0:55686:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 44 00000080:00200008:0.0:1787826423.530952:0:55686:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154400 00000002:00010000:0.0:1787826423.530967:0:55686:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114401 00010000:00010000:0.0:1787826423.531017:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab153e lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.531025:0:55686:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab153e lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826423.531037:0:55686:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826423.531049:0:55686:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aee580 x1874670563964672/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826423.531057:0:55686:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab153e lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826423.531077:0:55686:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563964672:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826423.537000:0:55686:0:(client.c:2715:ptlrpc_set_wait()) set ffff976891c9acc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826423.547733:0:55686:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563964672:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826423.547761:0:55686:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab153e lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b831b2 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.547776:0:55686:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab153e lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b831b2 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.547786:0:55686:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab153e lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b831b2 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826423.547799:0:55686:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff976882aee580 x1874670563964672/t8589935162(8589935162) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826423.547813:0:55686:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aee580 x1874670563964672/t8589935162(8589935162) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826423.547861:0:55686:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab153e lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b831b2 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826423.547874:0:55686:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab153e lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b831b2 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.547886:0:55686:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826423.547892:0:55686:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826423.547943:0:55686:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 31 -> 31 00000080:00010000:3.0:1787826423.547949:0:55686:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab153e 00000080:00010000:3.0:1787826423.547955:0:55686:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab153e from it ffffb2d442213bf0 00010000:00010000:3.0:1787826423.547959:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab153e lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b831b2 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.547969:0:55686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab153e lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b831b2 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000002:00100000:3.0:1787826423.547997:0:55686:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aee580 x1874670563964672/t8589935162(8589935162) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1787826423.548135:0:55686:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:3.0:1787826423.548143:0:55686:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:3.0:1787826423.548172:0:55686:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826423.548179:0:55686:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:3.0:1787826423.548209:0:55686:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826423.548220:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1530 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83188 expref: -99 pid: 55680 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826423.548232:0:55686:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1530 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83188 expref: -99 pid: 55680 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826423.548239:0:55686:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826423.548244:0:55686:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:3.0:1787826423.548247:0:55686:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1530 00000080:00010000:3.0:1787826423.548251:0:55686:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1530 from it ffffb2d442213cd8 00010000:00010000:3.0:1787826423.548255:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1530 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83188 expref: -99 pid: 55680 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826423.548347:0:55686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1530 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83188 expref: -99 pid: 55680 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826423.548466:0:55686:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 29 restore needed 0 00000001:00010000:3.0:1787826423.548476:0:55686:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:3.0:1787826423.548493:0:55686:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1787826423.548496:0:55686:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff9768837ffcb0, osc lock ffff9768984efd40, flags 0x400001002 00010000:00010000:3.0:1787826423.548511:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826423.548522:0:55686:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826423.548531:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826423.548538:0:55686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826423.548576:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007cfd3865/0x8e7c99ee62ab1545 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.548584:0:55686:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007cfd3865/0x8e7c99ee62ab1545 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787826423.548758:0:55686:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1787826423.552421:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826423.552513:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585277696 00000100:00100000:1.0:1787826423.552522:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585277696 00000100:00100000:1.0:1787826423.552582:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 531 00000100:00100000:1.0:1787826423.552591:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aec000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:30695:x1874670585277696:12345-192.168.203.139@tcp:106: 00010000:00010000:1.0:1787826423.552608:0:53061:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826423.552622:0:53061:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00010000:1.0:1787826423.552649:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@00000000349e195f[0x110, 1, [0x200000405:0x3a:0x0] hash] 00000020:00010000:1.0:1787826423.552651:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826422 blocks: 0 00000020:00010000:1.0:1787826423.552655:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@000000001fe7a5ce[0x116, 2, [0x280000401:0x10c:0x0]] 00000020:00010000:1.0:1787826423.552657:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826422 blocks: 0 00000100:00100000:1.0:1787826423.552709:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aec000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:30695:x1874670585277696:12345-192.168.203.139@tcp:106: Request processed in 119us (296us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826423.552716:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 531 00000080:00200000:3.0:1787826423.555921:0:55686:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:3.0:1787826423.555928:0:55686:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf04f0 (ffff976890d10128/0) at cl_lock_release():245 00000020:00010000:3.0:1787826423.555932:0:55686:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf04f0 (ffff976890d10128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826423.555939:0:55686:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768837ffcb0 (ffff976883b9d518/1) at cl_lock_cancel():142 00000020:00010000:3.0:1787826423.555942:0:55686:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf04f0 (ffff976890d10128/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826423.555946:0:55686:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768837ffcb0 (ffff976890d10128/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826423.555959:0:55686:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 31 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1787826423.556015:0:55686:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154000 00010000:00010000:3.0:1787826423.556138:0:55686:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab153e lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b831b2 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826423.556257:0:55686:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826423.556367:0:55686:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826423.556386:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826423.556395:0:55686:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826423.556402:0:55686:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826423.556409:0:55686:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826423.556413:0:55686:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826423.556418:0:55686:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d442213bf8 00010000:00010000:3.0:1787826423.556422:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826423.556429:0:55686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826423.556444:0:55686:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826423.556451:0:55686:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:3.0:1787826423.556456:0:55686:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787826423.556460:0:55686:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:3.0:1787826423.556464:0:55686:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 46 00000080:00200008:3.0:1787826423.556470:0:55686:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 1154402 00000002:00010000:3.0:1787826423.556484:0:55686:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114403 00010000:00010000:3.0:1787826423.556527:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab154c lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.556783:0:55686:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab154c lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787826423.556798:0:55686:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787826423.556806:0:55686:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff97688cb5d680 x1874670563965056/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787826423.556815:0:55686:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab154c lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826423.556940:0:55686:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563965056:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826423.557126:0:55686:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25300 going to sleep for 11 seconds 00000100:00100000:0.0:1787826423.572018:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826423.572156:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585277824 00000100:00100000:1.0:1787826423.572166:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585277824 00000100:00100000:1.0:1787826423.572228:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 532 00000100:00100000:1.0:1787826423.572238:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aee940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:8247:x1874670585277824:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826423.572256:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1530 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83188 expref: -99 pid: 55680 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826423.572352:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826423.572359:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aee940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:8247:x1874670585277824:12345-192.168.203.139@tcp:104: Request processed in 122us (350us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826423.572365:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 532 00000100:00100000:3.0:1787826423.588918:0:55686:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563965056:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826423.588948:0:55686:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode CW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab154c lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b831ce expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.588962:0:55686:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab154c lrc: 4/1,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b831ce expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.588974:0:55686:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab154c lrc: 4/1,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b831ce expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.588989:0:55686:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab154c lrc: 4/1,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b831ce expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.589002:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab154c lrc: 5/1,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b831ce expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.589010:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab154c lrc: 5/1,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b831ce expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.589017:0:55686:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab154c lrc: 4/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b831ce expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826423.589028:0:55686:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff97688cb5d680 x1874670563965056/t8589935163(8589935163) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826423.589042:0:55686:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff97688cb5d680 x1874670563965056/t8589935163(8589935163) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826423.589056:0:55686:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab154c lrc: 3/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b831ce expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.589070:0:55686:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 8 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826423.589076:0:55686:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1787826423.589120:0:55686:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab154c 00000080:00010000:3.0:1787826423.589127:0:55686:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab154c from it ffffb2d442213bf0 00010000:00010000:3.0:1787826423.589130:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab154c lrc: 3/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b831ce expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.589145:0:55686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab154c lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b831ce expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:3.0:1787826423.589173:0:55686:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff97688cb5d680 x1874670563965056/t8589935163(8589935163) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:3.0:1787826423.589264:0:55686:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=800466af arg=7ffd3631a0cc 00000080:00200000:3.0:1787826423.590461:0:55686:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826423.590486:0:55686:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826423.590502:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826423.590516:0:55686:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826423.590524:0:55686:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826423.590531:0:55686:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826423.590535:0:55686:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826423.590539:0:55686:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d442213bf8 00010000:00010000:3.0:1787826423.590543:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826423.590684:0:55686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826423.590698:0:55686:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826423.590705:0:55686:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=24 00000080:00200000:3.0:1787826423.590712:0:55686:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787826423.590727:0:55686:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787826423.590732:0:55686:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=. :VOLATILE:0000:490CF55C:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), file ffff97688773d980, open_flags 2f9c1, mode 8180 opened 0 00000080:00200000:3.0:1787826423.590739:0:55686:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=. :VOLATILE:0000:490CF55C:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=open|creat 00000002:00010000:3.0:1787826423.590759:0:55686:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: . :VOLATILE:0000:490CF55C:fd=03,[0x200000405:0x3c:0x0]) in obj [0x200000bd0:0x4:0x0], intent: open|creat flags 0574702 00010000:00010000:3.0:1787826423.590808:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1553 lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.590816:0:55686:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1553 lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787826423.590906:0:55686:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787826423.590916:0:55686:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff97688cb5cb40 x1874670563965440/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/66640 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787826423.590925:0:55686:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1553 lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826423.590944:0:55686:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563965440:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826423.591145:0:55686:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25300 going to sleep for 11 seconds 00000100:00100000:3.0:1787826423.601208:0:55686:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563965440:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826423.601235:0:55686:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1553 lrc: 4/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b831e3 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.601379:0:55686:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1553 lrc: 4/0,1 mode: --/PR res: [0x200000405:0x3c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b831e3 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.601396:0:55686:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1553 lrc: 4/0,1 mode: --/PR res: [0x200000405:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b831e3 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.601407:0:55686:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1553 lrc: 4/0,1 mode: PR/PR res: [0x200000405:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b831e3 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.601417:0:55686:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1553 lrc: 4/0,1 mode: PR/PR res: [0x200000405:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b831e3 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.601431:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1553 lrc: 5/1,1 mode: PR/PR res: [0x200000405:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b831e3 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.601439:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1553 lrc: 5/1,1 mode: PR/PR res: [0x200000405:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b831e3 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.601446:0:55686:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1553 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b831e3 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826423.601460:0:55686:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff97688cb5cb40 x1874670563965440/t8589935164(8589935164) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826423.601474:0:55686:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff97688cb5cb40 x1874670563965440/t8589935164(8589935164) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826423.601486:0:55686:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1553 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b831e3 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.601497:0:55686:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/60 (0 0) 00000080:00200000:3.0:1787826423.601503:0:55686:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd0:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1787826423.601516:0:55686:0:(namei.c:1628:ll_create_it()) VFS Op:name=. :VOLATILE:0000:490CF55C:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=open|creat 00000080:00200000:3.0:1787826423.601618:0:55686:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000405:0x3c:0x0](ffff97689017db88) 00000080:00200000:3.0:1787826423.601626:0:55686:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x3c:0x0], updating i_size 0 00000080:00200000:3.0:1787826423.601661:0:55686:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x3c:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787826423.601666:0:55686:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000405:0x3c:0x0](ffff97689017db88) 00000080:00010000:3.0:1787826423.601669:0:55686:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x3c:0x0](ffff97689017db88) for lock 0x8e7c99ee62ab1553 00000080:00010000:3.0:1787826423.601679:0:55686:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x3c:0x0](ffff97689017db88) 00000080:00200008:3.0:1787826423.601694:0:55686:0:(file.c:1090:ll_file_open()) START file . :VOLATILE:0000:490CF55C:fd=03:[0x200000405:0x3c:0x0](ffff97689017db88), flags 574701 00000002:00100000:3.0:1787826423.601716:0:55686:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff97688cb5cb40 x1874670563965440/t8589935164(8589935164) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1787826423.601749:0:55686:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1553 from it ffff9768869d97e0 00010000:00010000:3.0:1787826423.601754:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1553 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b831e3 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.601764:0:55686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1553 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b831e3 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826423.601891:0:55686:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x3c:0x0](ffff97689017db88), inode mode 8180 mask 40 00000080:00200000:3.0:1787826423.602149:0:55686:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x3c:0x0](ffff97689017db88), inode mode 8180 mask 2 00000080:00200000:3.0:1787826423.602159:0:55686:0:(xattr.c:414:ll_xattr_set()) VFS Op:inode=[0x200000405:0x3c:0x0](ffff97689017db88), xattr lov 00000002:00010000:3.0:1787826423.602176:0:55686:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x3c:0x0]) in obj [0x200000405:0x3c:0x0], intent: open flags 040000002 00010000:00010000:3.0:1787826423.602230:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab155a lrc: 3/1,0 mode: --/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.602238:0:55686:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab155a lrc: 3/1,0 mode: --/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787826423.602249:0:55686:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787826423.602256:0:55686:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff97688cb5da40 x1874670563965696/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787826423.602504:0:55686:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab155a lrc: 3/1,0 mode: --/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826423.602525:0:55686:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5da40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563965696:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826423.602759:0:55686:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25300 going to sleep for 11 seconds 00000100:00100000:0.0:1787826423.606681:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826423.606791:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585277952 00000100:00100000:1.0:1787826423.606801:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585277952 00000100:00100000:1.0:1787826423.607077:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 533 00000100:00100000:1.0:1787826423.607088:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aef480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6519:x1874670585277952:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826423.607106:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1553 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b831e3 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826423.607185:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826423.607192:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aef480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6519:x1874670585277952:12345-192.168.203.139@tcp:104: Request processed in 106us (520us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826423.607209:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 533 00000100:00100000:3.0:1787826423.613024:0:55686:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5da40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563965696:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826423.613042:0:55686:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab155a lrc: 4/1,0 mode: --/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.613055:0:55686:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab155a lrc: 4/1,0 mode: --/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.613063:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab155a lrc: 4/1,0 mode: --/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.613070:0:55686:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab155a lrc: 3/0,0 mode: --/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.613079:0:55686:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab155a lrc: 4/0,0 mode: --/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.613085:0:55686:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976888fb9800 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826423.613089:0:55686:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab155a lrc: 4/0,0 mode: --/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.613095:0:55686:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab155a lrc: 4/0,0 mode: --/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.613102:0:55686:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab155a lrc: 5/0,0 mode: --/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.613109:0:55686:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826423.613115:0:55686:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab155a lrc: 3/0,0 mode: --/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.613123:0:55686:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab155a lrc: 0/0,0 mode: --/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826423.613148:0:55686:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x800002b, status=0 req@ffff97688cb5da40 x1874670563965696/t8589935165(8589935165) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826423.613162:0:55686:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff97688cb5da40 x1874670563965696/t8589935165(8589935165) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1787826423.613180:0:55686:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x3c:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826423.613244:0:55686:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x3c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787826423.613253:0:55686:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x3c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826423.613516:0:55686:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x3c:0x0], updating i_size 0 00000002:00100000:3.0:1787826423.613672:0:55686:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff97688cb5da40 x1874670563965696/t8589935165(8589935165) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826423.613714:0:55686:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff97688cb5da40 x1874670563965696/t8589935165(8589935165) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826439 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826423.613725:0:55686:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1787826423.613730:0:55686:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff97688cb5c3c0 x1874670563966080/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826423.613751:0:55686:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563966080:192.168.203.139@tcp:35:lfs.0 00000100:00100000:3.0:1787826423.614028:0:55686:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25900 going to sleep for 11 seconds 00000100:00100000:3.0:1787826423.617331:0:55686:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563966080:192.168.203.139@tcp:35:lfs.0 00000002:00100000:3.0:1787826423.617365:0:55686:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff97688cb5da40 x1874670563965696/t8589935165(8589935165) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826439 ref 2 fl Complete:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1787826423.617415:0:55686:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826423.617423:0:55686:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:3.0:1787826423.617437:0:55686:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826423.617452:0:55686:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 30 res 8589935621/29 (0 0) 00010000:00010000:3.0:1787826423.617498:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab1561 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.617508:0:55686:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab1561 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.617517:0:55686:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab1561 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826423.617547:0:55686:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563966208:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826423.617661:0:55686:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25900 going to sleep for 11 seconds 00000100:00100000:1.0:1787826423.621614:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826423.622010:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585278080 00000100:00100000:1.0:1787826423.622020:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585278080 00000100:00100000:1.0:1787826423.622070:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 534 00000100:00100000:1.0:1787826423.622080:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0161680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:9966:x1874670585278080:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826423.622097:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab154c lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x22153cd396b831ce expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826423.622265:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826423.622273:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0161680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:9966:x1874670585278080:12345-192.168.203.139@tcp:104: Request processed in 195us (745us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826423.622282:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 534 00010000:00010000:3.0:1787826423.623504:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826423.623517:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab154c lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x22153cd396b831ce expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826423.623529:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976888fb8a00 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826423.623604:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab154c lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x22153cd396b831ce expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826423.623616:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab154c lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x22153cd396b831ce expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826423.623631:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab154c lrc: 3/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x22153cd396b831ce expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787826423.629117:0:55686:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563966208:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826423.629138:0:55686:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab1561 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8320d expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.629150:0:55686:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab1561 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8320d expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.629162:0:55686:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab1561 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8320d expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.629172:0:55686:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab1561 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8320d expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.629184:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab1561 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8320d expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.629193:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab1561 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8320d expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.629209:0:55686:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab1561 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8320d expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826423.629220:0:55686:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff97688cb5cf00 x1874670563966208/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 576/720 e 0 to 0 dl 1787826439 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826423.629233:0:55686:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab1561 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8320d expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826423.630217:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563966464:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826423.634155:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826423.634211:0:42211:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1787826423.634229:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585278208 00000100:00100000:1.0:1787826423.634238:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585278208 00000100:00100000:1.0:1787826423.634287:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 535 00000100:00100000:1.0:1787826423.634298:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff97688a33c850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:8436:x1874670585278208:12345-192.168.203.139@tcp:106: 00010000:00010000:1.0:1787826423.634314:0:42211:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787826423.634328:0:42211:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00000020:00010000:1.0:1787826423.634356:0:42211:0:(cl_object.c:322:cl_object_glimpse()) header@00000000349e195f[0x110, 1, [0x200000405:0x3a:0x0] hash] 00000020:00010000:1.0:1787826423.634358:0:42211:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826422 blocks: 0 00000020:00010000:1.0:1787826423.634363:0:42211:0:(cl_object.c:322:cl_object_glimpse()) header@000000001fe7a5ce[0x116, 2, [0x280000401:0x10c:0x0]] 00000020:00010000:1.0:1787826423.634364:0:42211:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826422 blocks: 0 00000100:00100000:1.0:1787826423.634432:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff97688a33c850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:8436:x1874670585278208:12345-192.168.203.139@tcp:106: Request processed in 136us (285us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826423.634440:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 535 00000080:00200000:0.0:1787826423.639525:0:55686:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787826423.639531:0:55686:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf0ed0 (ffff976890d10c68/0) at cl_lock_release():245 00000020:00010000:0.0:1787826423.639544:0:55686:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf0ed0 (ffff976890d10c68/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826423.639549:0:55686:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757c130 (ffff976890d10e18/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826423.639552:0:55686:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf0ed0 (ffff976890d10c68/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826423.639555:0:55686:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757c130 (ffff976890d10c68/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826423.639567:0:55686:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 31 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787826423.639588:0:55686:0:(file.c:6313:ll_getattr_dentry()) START file . :VOLATILE:0000:490CF55C:fd=03:[0x200000405:0x3c:0x0](ffff97689017db88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826423.639595:0:55686:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x3c:0x0](ffff97689017db88),name=. :VOLATILE:0000:490CF55C:fd=03 00000002:00010000:0.0:1787826423.639614:0:55686:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x3c:0x0]) in obj [0x200000405:0x3c:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826423.639663:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab156f lrc: 3/1,0 mode: --/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.639671:0:55686:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab156f lrc: 3/1,0 mode: --/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.639679:0:55686:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab156f lrc: 3/1,0 mode: --/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826423.639698:0:55686:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563966592:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826423.639787:0:55686:0:(client.c:2715:ptlrpc_set_wait()) set ffff976891c9acc0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826423.650948:0:55686:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563966592:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826423.650974:0:55686:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab156f lrc: 4/1,0 mode: --/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83222 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.650986:0:55686:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab156f lrc: 4/1,0 mode: --/PR res: [0x200000405:0x3c:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83222 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.651004:0:55686:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab156f lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x3c:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83222 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.651016:0:55686:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab156f lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x3c:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83222 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.651040:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab156f lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x3c:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83222 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.651048:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab156f lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x3c:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83222 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.651054:0:55686:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab156f lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x3c:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83222 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826423.651066:0:55686:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff976882aef840 x1874670563966592/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 576/720 e 0 to 0 dl 1787826439 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826423.651078:0:55686:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab156f lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x3c:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83222 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826423.651092:0:55686:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab156f lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x3c:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83222 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.651104:0:55686:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/60 (0 0) 00000080:00200000:0.0:1787826423.651112:0:55686:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000405:0x3c:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826423.651163:0:55686:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x3c:0x0]: layout version change: 4294967294 -> 1 00000080:00010000:0.0:1787826423.651169:0:55686:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x3c:0x0](ffff97689017db88) for lock 0x8e7c99ee62ab156f 00000080:00010000:0.0:1787826423.651174:0:55686:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x3c:0x0](ffff97689017db88) 00000080:00010000:0.0:1787826423.651177:0:55686:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab156f from it ffffb2d442213cd8 00010000:00010000:0.0:1787826423.651180:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab156f lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x3c:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83222 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.651189:0:55686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab156f lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x3c:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83222 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826423.651234:0:55686:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x3c:0x0] ignore/verify layout 0/1, layout version 31 restore needed 0 00000001:00010000:0.0:1787826423.651244:0:55686:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x3c:0x0] 00000008:00010000:0.0:1787826423.651261:0:55686:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826423.651265:0:55686:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e55730, osc lock ffff9768849a3e78, flags 0x400001002 00010000:00010000:0.0:1787826423.651310:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab1576 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.651320:0:55686:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab1576 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826423.651355:0:55686:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1787826423.651550:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670563966720:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826423.654801:0:2401:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff9768877a4800: committing for last_committed 17179869679 gen 1 00010000:00010000:0.0:1787826423.654877:0:2401:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab1576 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.654897:0:2401:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab1576 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83229 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.654911:0:2401:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab1576 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83229 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.654922:0:2401:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab1576 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83229 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826423.654937:0:2401:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab1576 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83229 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826423.654949:0:2401:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1787826423.655016:0:2401:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab1576 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83229 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.655091:0:2401:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab1576 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83229 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.655099:0:2401:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab1576 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83229 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.655106:0:2401:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab1576 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83229 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1787826423.655125:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670563966720:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826423.661101:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563967104:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826423.663900:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826423.664125:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585280256 00000100:00100000:0.0:1787826423.664136:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585280256 00000100:00100000:0.0:1787826423.664237:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 536 00000100:00100000:0.0:1787826423.664248:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0162d00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30398:x1874670585280256:12345-192.168.203.139@tcp:106: 00010000:00010000:0.0:1787826423.664264:0:42210:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826423.664281:0:42210:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00000020:00010000:0.0:1787826423.664309:0:42210:0:(cl_object.c:322:cl_object_glimpse()) header@00000000349e195f[0x110, 1, [0x200000405:0x3a:0x0] hash] 00000020:00010000:0.0:1787826423.664311:0:42210:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826422 blocks: 0 00000020:00010000:0.0:1787826423.664315:0:42210:0:(cl_object.c:322:cl_object_glimpse()) header@000000001fe7a5ce[0x116, 2, [0x280000401:0x10c:0x0]] 00000020:00010000:0.0:1787826423.664316:0:42210:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826422 blocks: 0 00000100:00100000:0.0:1787826423.664492:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0162d00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30398:x1874670585280256:12345-192.168.203.139@tcp:106: Request processed in 128us (559us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826423.664503:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 536 00000080:00200000:0.0:1787826423.666962:0:55686:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787826423.666969:0:55686:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf0ed0 (ffff976890d10c68/0) at cl_lock_release():245 00000020:00010000:0.0:1787826423.666973:0:55686:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf0ed0 (ffff976890d10c68/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826423.666978:0:55686:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757ccb0 (ffff976890d10e18/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826423.666983:0:55686:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf0ed0 (ffff976890d10c68/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826423.666986:0:55686:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757ccb0 (ffff976890d10c68/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826423.667001:0:55686:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 31 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787826423.667210:0:55686:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826423.667219:0:55686:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:0.0:1787826423.667238:0:55686:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826423.667255:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab1561 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8320d expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826423.667285:0:55686:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab1561 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8320d expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826423.667293:0:55686:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826423.667301:0:55686:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:0.0:1787826423.667304:0:55686:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1561 00000080:00010000:0.0:1787826423.667309:0:55686:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1561 from it ffffb2d442213cd8 00010000:00010000:0.0:1787826423.667313:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab1561 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b8320d expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826423.667320:0:55686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab1561 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8320d expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826423.667339:0:55686:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 31 restore needed 0 00000001:00010000:0.0:1787826423.667348:0:55686:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:0.0:1787826423.667363:0:55686:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826423.667366:0:55686:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e55830, osc lock ffff9768849a24e0, flags 0x400001002 00010000:00010000:0.0:1787826423.667386:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826423.667395:0:55686:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826423.667404:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826423.667416:0:55686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826423.667449:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab158b lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.667457:0:55686:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab158b lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826423.667492:0:55686:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1787826423.667502:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1787826423.667514:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670563967232:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826423.671489:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826423.671735:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585280384 00000100:00100000:0.0:1787826423.671745:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585280384 00000100:00100000:0.0:1787826423.671885:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 537 00000100:00100000:0.0:1787826423.671896:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0161680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:8435:x1874670585280384:12345-192.168.203.139@tcp:106: 00010000:00010000:0.0:1787826423.671915:0:42211:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826423.671929:0:42211:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00000020:00010000:0.0:1787826423.671958:0:42211:0:(cl_object.c:322:cl_object_glimpse()) header@00000000349e195f[0x110, 1, [0x200000405:0x3a:0x0] hash] 00000020:00010000:0.0:1787826423.671960:0:42211:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826422 blocks: 0 00000020:00010000:0.0:1787826423.671965:0:42211:0:(cl_object.c:322:cl_object_glimpse()) header@000000001fe7a5ce[0x116, 2, [0x280000401:0x10c:0x0]] 00000020:00010000:0.0:1787826423.671966:0:42211:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826422 blocks: 0 00000100:00100000:0.0:1787826423.672032:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0161680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:8435:x1874670585280384:12345-192.168.203.139@tcp:106: Request processed in 127us (540us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826423.672040:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 537 00010000:00010000:0.0:1787826423.676634:0:2401:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab158b lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.676656:0:2401:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab158b lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.676667:0:2401:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab158b lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.676675:0:2401:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab158b lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.676683:0:2401:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab158b lrc: 5/1,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826423.676697:0:2401:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:0.0:1787826423.676704:0:2401:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1787826423.676759:0:2401:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab158b lrc: 4/1,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.676769:0:2401:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab158b lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.676783:0:2401:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab158b lrc: 4/0,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.676791:0:2401:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925c800 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:0.0:1787826423.676796:0:2401:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab158b lrc: 5/0,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.676808:0:2401:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787826423.676894:0:2401:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab158b lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.676904:0:2401:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab158b lrc: 0/0,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826423.676924:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670563967232:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826423.677804:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563967360:192.168.203.139@tcp:1:lfs.0 00000100:00100000:1.0:1787826423.682400:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826423.682480:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585280512 00000100:00100000:0.0:1787826423.682490:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585280512 00000100:00100000:0.0:1787826423.682544:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 538 00000100:00100000:0.0:1787826423.682554:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b01603c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30696:x1874670585280512:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826423.682573:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab14e3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x22153cd396b8307e expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826423.682651:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826423.682658:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b01603c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30696:x1874670585280512:12345-192.168.203.139@tcp:104: Request processed in 106us (278us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826423.682666:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 538 00000080:00200000:0.0:1787826423.690045:0:55686:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:0.0:1787826423.690056:0:55686:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:0.0:1787826423.690084:0:55686:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826423.690091:0:55686:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:0.0:1787826423.690109:0:55686:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826423.690127:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab1561 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8320d expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826423.690140:0:55686:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab1561 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8320d expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826423.690148:0:55686:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826423.690156:0:55686:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:0.0:1787826423.690159:0:55686:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1561 00000080:00010000:0.0:1787826423.690166:0:55686:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1561 from it ffffb2d442213cd8 00010000:00010000:0.0:1787826423.690170:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab1561 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b8320d expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826423.690179:0:55686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab1561 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8320d expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1787826423.690211:0:55686:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 31 restore needed 0 00000001:00010000:0.0:1787826423.690220:0:55686:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:0.0:1787826423.690236:0:55686:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826423.690240:0:55686:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e551b0, osc lock ffff9768849a2750, flags 0x400001002 00010000:00010000:0.0:1787826423.690284:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab1592 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.690295:0:55686:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab1592 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826423.690322:0:55686:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1787826423.690512:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1787826423.690526:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563967616:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826423.693670:0:2402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab1592 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.693696:0:2402:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab1592 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8325a expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.693709:0:2402:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab1592 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8325a expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.693720:0:2402:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab1592 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8325a expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826423.693735:0:2402:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab1592 lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b8325a expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826423.693746:0:2402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1787826423.693764:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab1592 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b8325a expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.693780:0:2402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab1592 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b8325a expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.693793:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab1592 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b8325a expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.693802:0:2402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab1592 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b8325a expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1787826423.693820:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563967616:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826423.755950:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826423.756129:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585280768 00000100:00100000:0.0:1787826423.756139:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585280768 00000100:00100000:0.0:1787826423.756192:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 539 00000100:00100000:0.0:1787826423.756212:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0163c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:8437:x1874670585280768:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826423.756230:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab14ea lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000030000 nid: local remote: 0x22153cd396b8308c expref: -99 pid: 55670 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826423.756317:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826423.756323:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0163c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:8437:x1874670585280768:12345-192.168.203.139@tcp:104: Request processed in 114us (382us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826423.756330:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 539 00000100:00100000:3.0:1787826423.756742:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563967872:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826423.762913:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563967872:192.168.203.139@tcp:103:ldlm_bl.0 00020000:00200000:3.0:1787826423.763381:0:55686:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826423.768947:0:55686:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x3c:0x0] updating i_size 0 i_blocks 0 00000080:00200000:3.0:1787826423.768963:0:55686:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x3c:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200080:3.0:1787826423.777488:0:55686:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=801066da arg=7ffd3631c110 00000080:00200000:3.0:1787826423.777512:0:55686:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826423.777525:0:55686:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1787826423.777613:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563968128:192.168.203.139@tcp:1:lfs.0 00000100:00100000:3.0:1787826423.782654:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563968128:192.168.203.139@tcp:1:lfs.0 00000080:00200008:0.0:1787826423.782774:0:55686:0:(llite_lib.c:2427:ll_setattr_raw()) START file . :VOLATILE:0000:490CF55C:fd=03:[0x200000405:0x3c:0x0](ffff97689017db88) current size 0, valid attrs 101f0, mode 9768, uid 0, gid 0, new size 4294967296, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826423.000000000 00010000:00010000:0.0:1787826423.782812:0:55686:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab156f lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x3c:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b83222 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826423.782859:0:55686:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x3c:0x0]: losing layout lock 00010000:00010000:0.0:1787826423.782905:0:55686:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab156f lrc: 1/0,0 mode: --/PR res: [0x200000405:0x3c:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b83222 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826423.782912:0:55686:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826423.782914:0:55686:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab156f lrc: 0/0,0 mode: --/PR res: [0x200000405:0x3c:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b83222 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1787826423.782938:0:55686:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787826423.782946:0:55686:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aef840 x1874670563968384/t0(0) o36->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826423.782968:0:55686:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563968384:192.168.203.139@tcp:36:lfs.0 00000100:00100000:0.0:1787826423.783041:0:55686:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e6c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826423.788648:0:55686:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563968384:192.168.203.139@tcp:36:lfs.0 00000080:00200000:0.0:1787826423.788764:0:55686:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x3c:0x0] ignore/verify layout 0/0, layout version 31 restore needed 0 00000080:00010000:0.0:1787826423.788870:0:55686:0:(file.c:7076:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000405:0x3c:0x0](ffff97689017db88) 00000002:00010000:0.0:1787826423.788881:0:55686:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x3c:0x0]) in obj [0x200000405:0x3c:0x0], intent: layout flags 0 00010000:00010000:0.0:1787826423.789005:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab1599 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.789016:0:55686:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab1599 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.789144:0:55686:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab1599 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826423.789165:0:55686:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed2c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563968512:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826423.789319:0:55686:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e6c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826423.795182:0:55686:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed2c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563968512:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826423.795226:0:55686:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab1599 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83276 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.795240:0:55686:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab1599 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83276 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826423.795253:0:55686:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff976882aed2c0 x1874670563968512/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 376/480 e 0 to 0 dl 1787826439 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826423.795264:0:55686:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab1599 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83276 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826423.795277:0:55686:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab1599 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83276 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.795289:0:55686:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/60 (0 0) 00000080:00200000:0.0:1787826423.795296:0:55686:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000405:0x3c:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1787826423.795312:0:55686:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x3c:0x0](ffff97689017db88) for lock 0x8e7c99ee62ab1599 00000080:00010000:0.0:1787826423.795319:0:55686:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1599 from it ffffb2d442213a78 00010000:00010000:0.0:1787826423.795323:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab1599 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83276 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.795343:0:55686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab1599 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83276 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826423.795370:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab1599 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83276 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826423.795382:0:55686:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab1599 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83276 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826423.795391:0:55686:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x3c:0x0](00000000229a7def) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab1599 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83276 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826423.795426:0:55686:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x3c:0x0]: layout version change: 4294967294 -> 1 00010000:00010000:0.0:1787826423.795431:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab1599 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83276 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826423.795439:0:55686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab1599 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83276 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:0.0:1787826423.795459:0:55686:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826423.796173:0:55686:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:0.0:1787826423.796190:0:55686:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787826423.796204:0:55686:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e55030, osc lock ffff9768849a2ea0, flags 0x0 00010000:00010000:0.0:1787826423.796218:0:55686:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9768ad0ae800 type 11 mode 2 res 11811161089/269 (0 18446744073709551615) 00010000:00010000:0.0:1787826423.796253:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.796263:0:55686:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.796273:0:55686:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826423.796298:0:55686:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed2c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563968640:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826423.796377:0:55686:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e540 going to sleep for 11 seconds 00000100:00100000:1.0:1787826423.804429:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826423.804653:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585280896 00000100:00100000:0.0:1787826423.804663:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585280896 00000100:00100000:1.0:1787826423.805652:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 540 00000100:00100000:1.0:1787826423.805896:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0162580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:14864:x1874670585280896:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826423.807017:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab1576 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b83229 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826423.807173:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826423.807184:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0162580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:14864:x1874670585280896:12345-192.168.203.139@tcp:104: Request processed in 1519us (2861us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826423.807202:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 540 00000100:00100000:3.0:1787826423.810221:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563968768:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826423.813537:0:55686:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed2c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563968640:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826423.813568:0:55686:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 4/0,1 mode: --/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.813736:0:55686:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 4/0,1 mode: --/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.813747:0:55686:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 4/0,1 mode: --/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826423.816991:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826423.817091:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585281024 00000100:00100000:0.0:1787826423.817101:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585281024 00000100:00100000:0.0:1787826423.817206:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 541 00000100:00100000:0.0:1787826423.817216:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0163840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:10340:x1874670585281024:12345-192.168.203.139@tcp:105: 00010000:00010000:0.0:1787826423.817232:0:42211:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 5/0,1 mode: --/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.817246:0:42211:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 5/0,1 mode: --/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.817256:0:42211:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 5/0,1 mode: --/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.817268:0:42211:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.817279:0:42211:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 6/0,1 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.817288:0:42211:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 6/0,1 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.817503:0:42211:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff97688925dc00) 00000100:00100000:0.0:1787826423.817564:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0163840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:10340:x1874670585281024:12345-192.168.203.139@tcp:105: Request processed in 348us (581us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826423.817575:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 541 00010000:00010000:3.0:1787826423.817778:0:55686:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.817791:0:55686:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.817808:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826423.817843:0:55686:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 4/0,2 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.817858:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.817867:0:55686:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:0.0:1787826423.821673:0:55686:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976890fb7128 (ffff976890d10c68/0) at cl_lock_release():245 00000020:00010000:0.0:1787826423.821679:0:55686:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976890fb7128 (ffff976890d10c68/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826423.821685:0:55686:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e55030 (ffff976890d10128/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787826423.821694:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.821709:0:55686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787826423.821724:0:55686:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976890fb7128 (ffff976890d10c68/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826423.821727:0:55686:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e55030 (ffff976890d10c68/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826423.821743:0:55686:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x3c:0x0] ignore/verify layout 0/0, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787826423.821758:0:55686:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file . :VOLATILE:0000:490CF55C:fd=03:[0x200000405:0x3c:0x0](ffff97689017db88) current size 0, valid attrs 100101f0, mode 9768, uid 0, gid 0, new size 4294967296, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826423.000000000, rc 0 00000080:00200008:0.0:1787826423.821905:0:55686:0:(llite_lib.c:2427:ll_setattr_raw()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8) current size 0, valid attrs 101f0, mode 9768, uid 0, gid 0, new size 0, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826423.000000000 00010000:00010000:0.0:1787826423.821938:0:55686:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab1561 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b8320d expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826423.821961:0:55686:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab153e lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b831b2 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826423.822006:0:55686:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab1561 lrc: 1/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b8320d expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826423.822018:0:55686:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826423.822021:0:55686:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab1561 lrc: 0/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b8320d expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000020:00100000:0.0:1787826423.822041:0:55686:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787826423.822051:0:55686:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aee1c0 x1874670563969152/t0(0) o36->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826423.822072:0:55686:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee1c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563969152:192.168.203.139@tcp:36:lfs.0 00000100:00100000:0.0:1787826423.822207:0:55686:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e780 going to sleep for 11 seconds 00000100:00100000:3.0:1787826423.822325:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563968768:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826423.826091:0:55686:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee1c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563969152:192.168.203.139@tcp:36:lfs.0 00000080:00200000:0.0:1787826423.826134:0:55686:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 1 restore needed 0 00020000:00200000:0.0:1787826423.826150:0:55686:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826423.826163:0:55686:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:0.0:1787826423.826179:0:55686:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787826423.826183:0:55686:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e55030, osc lock ffff9768849a2ea0, flags 0x0 00010000:00010000:0.0:1787826423.826211:0:55686:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976889533000 type 11 mode 2 res 10737419265/268 (0 18446744073709551615) 00010000:00010000:0.0:1787826423.826255:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab15a7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.826268:0:55686:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab15a7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.826278:0:55686:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab15a7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826423.826299:0:55686:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563969280:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826423.826490:0:55686:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e780 going to sleep for 11 seconds 00000100:00100000:1.0:1787826423.828820:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826423.829053:0:42210:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:0.0:1787826423.829072:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585281152 00000100:00100000:0.0:1787826423.829081:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585281152 00000100:00100000:0.0:1787826423.829133:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 542 00000100:00100000:0.0:1787826423.829142:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff97688a33c050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:8436:x1874670585281152:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826423.829159:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab1592 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b8325a expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826423.829234:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826423.829240:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff97688a33c050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:8436:x1874670585281152:12345-192.168.203.139@tcp:104: Request processed in 100us (429us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826423.829247:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 542 00000100:00100000:1.0:1787826423.834169:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826423.834343:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585281280 00000100:00100000:1.0:1787826423.834352:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585281280 00000100:00100000:1.0:1787826423.834406:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 543 00000100:00100000:1.0:1787826423.834416:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0161e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:6505:x1874670585281280:12345-192.168.203.139@tcp:105: 00010000:00010000:1.0:1787826423.834432:0:42211:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab15a7 lrc: 4/0,1 mode: --/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8328b expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.834444:0:42211:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab15a7 lrc: 4/0,1 mode: --/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8328b expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.834454:0:42211:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab15a7 lrc: 4/0,1 mode: --/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8328b expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.834466:0:42211:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab15a7 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000000000 nid: local remote: 0x22153cd396b8328b expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.834475:0:42211:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab15a7 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000000000 nid: local remote: 0x22153cd396b8328b expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.834484:0:42211:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab15a7 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000000 nid: local remote: 0x22153cd396b8328b expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.834501:0:42211:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff97688925da00) 00000100:00100000:1.0:1787826423.834554:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0161e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:6505:x1874670585281280:12345-192.168.203.139@tcp:105: Request processed in 139us (394us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826423.834561:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 543 00000100:00100000:0.0:1787826423.835546:0:55686:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563969280:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826423.835569:0:55686:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab15a7 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8328b expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.835660:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab15a7 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b8328b expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826423.835676:0:55686:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab15a7 lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x22153cd396b8328b expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.835689:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab15a7 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8328b expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826423.835702:0:55686:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab15a7 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8328b expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826423.835943:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563969536:192.168.203.139@tcp:2:lfs.0 00000100:00100000:3.0:1787826423.837097:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563969408:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826423.849210:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563969536:192.168.203.139@tcp:2:lfs.0 00000020:00010000:3.0:1787826423.849292:0:55686:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976890fb7128 (ffff976890d10c68/0) at cl_lock_release():245 00000020:00010000:3.0:1787826423.849297:0:55686:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976890fb7128 (ffff976890d10c68/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826423.849302:0:55686:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e55030 (ffff976890d10128/1) at cl_lock_cancel():142 00010000:00010000:3.0:1787826423.849312:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab15a7 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8328b expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.849328:0:55686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab15a7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8328b expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:3.0:1787826423.849344:0:55686:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976890fb7128 (ffff976890d10c68/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826423.849347:0:55686:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e55030 (ffff976890d10c68/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826423.849363:0:55686:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 31 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1787826423.849392:0:55686:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8) current size 0, valid attrs 100101f0, mode 9768, uid 0, gid 0, new size 0, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826423.000000000, rc 0 00000080:00200080:3.0:1787826423.849418:0:55686:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=e0e570 00010000:00010000:3.0:1787826423.849467:0:55686:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab157d lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0x22153cd396b83237 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.849485:0:55686:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab157d lrc: 0/0,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x85809400000000 nid: local remote: 0x22153cd396b83237 expref: -99 pid: 55686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826423.849510:0:55686:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889efb840 x1874670563966976/t8589935167(8589935167) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826439 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826423.849521:0:55686:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1787826423.849527:0:55686:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff97688cb5de00 x1874670563969792/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 504/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826423.849547:0:55686:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563969792:192.168.203.139@tcp:35:lfs.0 00000100:00100000:3.0:1787826423.849624:0:55686:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25900 going to sleep for 11 seconds 00000100:00100000:1.0:1787826423.861298:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826423.861342:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826423.861370:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585281408 00000100:00100000:1.0:1787826423.861380:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585281408 00000100:00100000:1.0:1787826423.861422:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585281536 00000100:00100000:1.0:1787826423.861425:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585281536 00000100:00100000:1.0:1787826423.861441:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 544 00000100:00100000:1.0:1787826423.861450:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0160f00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+6:6521:x1874670585281408:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826423.861469:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab153e lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b831b2 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1787826423.861545:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826423.861552:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0160f00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+6:6521:x1874670585281408:12345-192.168.203.139@tcp:104: Request processed in 103us (265us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826423.861559:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 544 00000100:00100000:1.0:1787826423.861577:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 545 00000100:00100000:1.0:1787826423.861583:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0162d00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6521:x1874670585281536:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826423.861591:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf400/0x8e7c99ee62ab1529 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83165 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826423.861626:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:1.0:1787826423.861632:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0162d00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6521:x1874670585281536:12345-192.168.203.139@tcp:104: Request processed in 49us (290us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826423.861638:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 545 00010000:00010000:3.0:1787826423.861934:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1787826423.861942:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab153e lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b831b2 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826423.861951:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925c400 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826423.861956:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab153e lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b831b2 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826423.861965:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab153e lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b831b2 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826423.861973:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab153e lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 3 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b831b2 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826423.861997:0:42220:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf400/0x8e7c99ee62ab1529 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83165 expref: -99 pid: 55678 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000002:00100000:3.0:1787826423.862056:0:42220:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976882aee580 x1874670563964672/t8589935162(8589935162) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826439 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826423.862069:0:42220:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 2 is allocated opc 35, max 7 00000100:00100000:3.0:1787826423.862074:0:42220:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff97688cb5f840 x1874670563970048/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826423.862096:0:42220:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:42220:1874670563970048:192.168.203.139@tcp:35:ldlm_bl.0 00000100:00100000:3.0:1787826423.862153:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cac6780 going to sleep for 11 seconds 00000100:00100000:3.0:1787826423.867619:0:42220:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:42220:1874670563970048:192.168.203.139@tcp:35:ldlm_bl.0 00000002:00100000:3.0:1787826423.867658:0:42220:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976882aee580 x1874670563964672/t8589935162(8589935162) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826439 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1787826423.867730:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab153e lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b831b2 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826423.867740:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826423.872002:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab153e lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b831b2 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826423.872033:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab153e lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b831b2 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826423.872062:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826423.872065:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826423.872068:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826423.880175:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826423.880273:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585281664 00000100:00100000:0.0:1787826423.880282:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585281664 00000100:00100000:0.0:1787826423.880338:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 546 00000100:00100000:0.0:1787826423.880347:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0161680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6521:x1874670585281664:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826423.880366:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab1599 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x3c:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83276 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826423.880439:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826423.880446:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0161680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6521:x1874670585281664:12345-192.168.203.139@tcp:104: Request processed in 100us (276us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826423.880453:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 546 00000100:00100000:3.0:1787826423.911729:0:55686:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563969792:192.168.203.139@tcp:35:lfs.0 00000002:00100000:3.0:1787826423.911773:0:55686:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889efb840 x1874670563966976/t8589935167(8589935167) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826439 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1787826423.911856:0:55686:0:(file.c:7076:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000002:00010000:3.0:1787826423.911868:0:55686:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: layout flags 0 00010000:00010000:3.0:1787826423.911913:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.911922:0:55686:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.911931:0:55686:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55686 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826423.911948:0:55686:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563970560:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826423.912040:0:55686:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cac6780 going to sleep for 11 seconds 00000100:00100000:3.0:1787826423.919289:0:55686:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563970560:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826423.919321:0:55686:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b832a0 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.919335:0:55686:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b832a0 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826423.919349:0:55686:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff97688cb5de00 x1874670563970560/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 376/584 e 0 to 0 dl 1787826439 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826423.919359:0:55686:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b832a0 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826423.919370:0:55686:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b832a0 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.919397:0:55686:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826423.919403:0:55686:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1787826423.919420:0:55686:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab15ae 00000080:00010000:3.0:1787826423.919428:0:55686:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab15ae from it ffffb2d442213bc8 00010000:00010000:3.0:1787826423.919431:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b832a0 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.919440:0:55686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b832a0 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826423.919462:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b832a0 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826423.919481:0:55686:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b832a0 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1787826423.919489:0:55686:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b832a0 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826423.919524:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b832a0 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826423.919531:0:55686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b832a0 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826423.919548:0:55686:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 31 restore needed 0 00020000:00200000:3.0:1787826423.919560:0:55686:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826423.919585:0:55686:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:3.0:1787826423.919596:0:55686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b832a0 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826423.919603:0:55686:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b832a0 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:3.0:1787826423.919610:0:55686:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b832a0 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826423.919626:0:55686:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826423.919630:0:55686:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826423.919637:0:55686:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000008:00010000:3.0:1787826423.919649:0:55686:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 0 [cma]time: 1787826422 1787826412 1787826412 by oinfo size 0 blocks 0 [cma]time 1787826423 1787826412 1787826412 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab15a7 lrc: 1/0,0 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8328b expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826423.919706:0:55686:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 4294967294 -> 32 00010000:00010000:3.0:1787826423.919711:0:55686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b832a0 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826423.919719:0:55686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b832a0 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:3.0:1787826423.919747:0:55686:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 1154002 00000002:00100000:3.0:1787826423.919787:0:55686:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff97688cb5d680 x1874670563965056/t8589935163(8589935163) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826439 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826423.919798:0:55686:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1787826423.919803:0:55686:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff97688cb5de00 x1874670563970816/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826423.919821:0:55686:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563970816:192.168.203.139@tcp:35:lfs.0 00000100:00100000:3.0:1787826423.919947:0:55686:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cac6780 going to sleep for 11 seconds 00000100:00100000:3.0:1787826423.931925:0:55686:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563970816:192.168.203.139@tcp:35:lfs.0 00000002:00100000:3.0:1787826423.931965:0:55686:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff97688cb5d680 x1874670563965056/t8589935163(8589935163) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826439 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1787826423.932047:0:55686:0:(file.c:479:ll_file_release()) START file . :VOLATILE:0000:490CF55C:fd=03:[0x200000405:0x3c:0x0](ffff97689017db88), flags 554001 00000002:00100000:3.0:1787826423.932093:0:55686:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff97688cb5cb40 x1874670563965440/t8589935164(8589935164) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826439 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826423.932104:0:55686:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1787826423.932110:0:55686:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff97688cb5de00 x1874670563971072/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826423.932133:0:55686:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563971072:192.168.203.139@tcp:35:lfs.0 00000100:00100000:3.0:1787826423.932228:0:55686:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cac6780 going to sleep for 11 seconds 00000100:00100000:3.0:1787826423.946186:0:55686:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55686:1874670563971072:192.168.203.139@tcp:35:lfs.0 00000002:00100000:3.0:1787826423.946232:0:55686:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff97688cb5cb40 x1874670563965440/t8589935164(8589935164) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826439 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1787826423.963700:0:55689:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826423.963730:0:55689:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826423.963749:0:55689:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826423.963764:0:55689:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826423.963773:0:55689:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826423.963780:0:55689:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826423.963784:0:55689:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826423.963791:0:55689:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4421e3b58 00010000:00010000:3.0:1787826423.963796:0:55689:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826423.963805:0:55689:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826423.963820:0:55689:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826423.963856:0:55689:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=24 00000080:00200000:3.0:1787826423.963862:0:55689:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787826423.963872:0:55689:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), flags=8 00000080:00200000:3.0:1787826423.963876:0:55689:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=getattr 00000002:00010000:3.0:1787826423.963889:0:55689:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f60b.sanity-flr,[0x0:0x0:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 01 00010000:00010000:3.0:1787826423.963938:0:55689:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab15b5 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55689 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.963947:0:55689:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab15b5 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55689 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.963957:0:55689:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab15b5 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55689 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826423.963980:0:55689:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55689:1874670563971200:192.168.203.139@tcp:101:getfattr.0 00000100:00100000:3.0:1787826423.964214:0:55689:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cac6780 going to sleep for 11 seconds 00000100:00100000:3.0:1787826423.978993:0:55689:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55689:1874670563971200:192.168.203.139@tcp:101:getfattr.0 00010000:00010000:3.0:1787826423.979486:0:55689:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab15b5 lrc: 4/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b832c3 expref: -99 pid: 55689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.979509:0:55689:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab15b5 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b832c3 expref: -99 pid: 55689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.979518:0:55689:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab15b5 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b832c3 expref: -99 pid: 55689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.979530:0:55689:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab15b5 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b832c3 expref: -99 pid: 55689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.979542:0:55689:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab15b5 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b832c3 expref: -99 pid: 55689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.979556:0:55689:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab15b5 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b832c3 expref: -99 pid: 55689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.979564:0:55689:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab15b5 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b832c3 expref: -99 pid: 55689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826423.979572:0:55689:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab15b5 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b832c3 expref: -99 pid: 55689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826423.979584:0:55689:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff97688cb5cb40 x1874670563971200/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 584/824 e 0 to 0 dl 1787826439 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'getfattr.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826423.979595:0:55689:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab15b5 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b832c3 expref: -99 pid: 55689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787826423.982106:0:55689:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab15b5 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b832c3 expref: -99 pid: 55689 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.982135:0:55689:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/29 (0 0) 00000080:00200000:1.0:1787826423.982142:0:55689:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787826423.982190:0:55689:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x1d:0x0], updating i_size 0 00000080:00200000:1.0:1787826423.982203:0:55689:0:(namei.c:147:ll_iget()) got inode: [0x200000405:0x1d:0x0](ffff9768901a80c8): rc = 0 00000080:00200000:1.0:1787826423.982218:0:55689:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 32 -> 32 00000080:00010000:1.0:1787826423.982222:0:55689:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab15b5 00000080:00010000:1.0:1787826423.982233:0:55689:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:1.0:1787826423.982236:0:55689:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab15b5 from it ffffb2d4421e3b20 00010000:00010000:1.0:1787826423.982239:0:55689:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab15b5 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b832c3 expref: -99 pid: 55689 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826423.982249:0:55689:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab15b5 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b832c3 expref: -99 pid: 55689 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1787826423.982289:0:55689:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1787826423.982296:0:55689:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:1.0:1787826423.982321:0:55689:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826423.982333:0:55689:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab15b5 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b832c3 expref: -99 pid: 55689 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826423.982344:0:55689:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab15b5 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b832c3 expref: -99 pid: 55689 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787826423.982351:0:55689:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826423.982359:0:55689:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:1.0:1787826423.982362:0:55689:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab15b5 00000080:00010000:1.0:1787826423.982482:0:55689:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab15b5 from it ffffb2d4421e3cb8 00010000:00010000:1.0:1787826423.982486:0:55689:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab15b5 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b832c3 expref: -99 pid: 55689 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826423.982493:0:55689:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab15b5 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b832c3 expref: -99 pid: 55689 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787826423.982532:0:55689:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826423.982541:0:55689:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826423.982548:0:55689:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826423.982556:0:55689:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826423.982563:0:55689:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826423.982567:0:55689:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826423.982569:0:55689:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826423.982573:0:55689:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4421e3bf8 00010000:00010000:1.0:1787826423.982576:0:55689:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826423.982584:0:55689:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826423.982591:0:55689:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826423.982596:0:55689:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:1.0:1787826423.982601:0:55689:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826423.982604:0:55689:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=9 00000080:00200000:1.0:1787826423.982611:0:55689:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:1.0:1787826423.982617:0:55689:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200000:1.0:1787826423.982632:0:55689:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826423.982639:0:55689:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826423.982645:0:55689:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826423.982652:0:55689:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826423.982658:0:55689:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826423.982662:0:55689:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826423.982664:0:55689:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826423.982668:0:55689:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4421e3bf8 00010000:00010000:1.0:1787826423.982671:0:55689:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826423.982678:0:55689:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826423.982686:0:55689:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826423.982689:0:55689:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:1.0:1787826423.982693:0:55689:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826423.982695:0:55689:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=9 00000080:00200000:1.0:1787826423.982702:0:55689:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:1.0:1787826423.982705:0:55689:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200000:1.0:1787826424.004599:0:55693:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826424.004632:0:55693:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826424.004652:0:55693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826424.004669:0:55693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826424.004676:0:55693:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826424.004684:0:55693:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826424.004687:0:55693:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826424.004695:0:55693:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44220bbf8 00010000:00010000:1.0:1787826424.004699:0:55693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826424.004707:0:55693:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826424.004723:0:55693:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826424.004731:0:55693:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=27 00000080:00200000:1.0:1787826424.004736:0:55693:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826424.004740:0:55693:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=267 00000080:00200000:1.0:1787826424.004765:0:55693:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826424.004773:0:55693:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826424.004780:0:55693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826424.004788:0:55693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826424.004795:0:55693:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826424.004798:0:55693:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826424.004801:0:55693:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826424.004805:0:55693:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44220bbf8 00010000:00010000:1.0:1787826424.004808:0:55693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826424.004816:0:55693:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826424.004941:0:55693:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826424.004946:0:55693:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=267 00000080:00200000:1.0:1787826424.004949:0:55693:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 44 00000080:00200008:1.0:1787826424.004955:0:55693:0:(file.c:1090:ll_file_open()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00000080:00200080:1.0:1787826424.004984:0:55693:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8) cmd=c0086915 arg=aa23d0 00000100:00100000:1.0:1787826424.005051:0:55693:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b01603c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55693:1874670563971328:192.168.203.139@tcp:34:lfs.0 00000100:00100000:1.0:1787826424.005155:0:55693:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3300 going to sleep for 11 seconds 00000100:00100000:0.0:1787826424.008992:0:55693:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b01603c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55693:1874670563971328:192.168.203.139@tcp:34:lfs.0 00000080:00200008:0.0:1787826424.010116:0:55693:0:(file.c:479:ll_file_release()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00010000:00010000:0.0:1787826424.010154:0:55693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9600/0x8e7c99ee62ab0c1c lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b818c9 expref: -99 pid: 55483 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:1.0:1787826424.026934:0:55695:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826424.026969:0:55695:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826424.026988:0:55695:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826424.027004:0:55695:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826424.027012:0:55695:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826424.027019:0:55695:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826424.027023:0:55695:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826424.027030:0:55695:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d442213bf8 00010000:00010000:1.0:1787826424.027034:0:55695:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826424.027043:0:55695:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826424.027061:0:55695:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826424.027070:0:55695:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:1.0:1787826424.027077:0:55695:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826424.027081:0:55695:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:1.0:1787826424.027085:0:55695:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 44 00000080:00200008:1.0:1787826424.027092:0:55695:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154400 00000002:00010000:1.0:1787826424.027108:0:55695:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114401 00010000:00010000:1.0:1787826424.027162:0:55695:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab15bc lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 55695 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826424.027171:0:55695:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab15bc lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55695 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1787826424.027183:0:55695:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1787826424.027206:0:55695:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b0163c00 x1874670563971584/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1787826424.027215:0:55695:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab15bc lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55695 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826424.027239:0:55695:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0163c00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55695:1874670563971584:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826424.027593:0:55695:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3480 going to sleep for 11 seconds 00000100:00100000:1.0:1787826424.037248:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826424.037339:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585282176 00000100:00100000:0.0:1787826424.037350:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585282176 00000100:00100000:0.0:1787826424.037426:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 547 00000100:00100000:0.0:1787826424.037437:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0162580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:8247:x1874670585282176:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826424.037455:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9e00/0x8e7c99ee62ab15b5 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b832c3 expref: -99 pid: 55689 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826424.037535:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826424.037542:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0162580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:8247:x1874670585282176:12345-192.168.203.139@tcp:104: Request processed in 107us (305us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826424.037550:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 547 00000100:00100000:3.0:1787826424.037943:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563971968:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826424.040639:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563971968:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826424.057185:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826424.057281:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585282304 00000100:00100000:1.0:1787826424.057291:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585282304 00000100:00100000:1.0:1787826424.057348:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 548 00000100:00100000:1.0:1787826424.057357:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aec780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6522:x1874670585282304:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826424.057390:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b832a0 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787826424.057468:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826424.057475:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aec780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6522:x1874670585282304:12345-192.168.203.139@tcp:104: Request processed in 120us (299us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826424.057482:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 548 00010000:00010000:3.0:1787826424.057875:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826424.057883:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b832a0 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826424.057892:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976888fb9800 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826424.057896:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b832a0 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826424.057905:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b832a0 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826424.057913:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b832a0 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1787826424.057935:0:42220:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x1d:0x0]: losing layout lock 00010000:00010000:3.0:1787826424.057973:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b832a0 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826424.057980:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826424.058008:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b832a0 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826424.058016:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab15ae lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b832a0 expref: -99 pid: 55686 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826424.058041:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.058043:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.058044:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826424.058051:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826424.058067:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563972992:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826424.065111:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563972992:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826424.086549:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826424.086654:0:42211:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:0.0:1787826424.086673:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585282432 00000100:00100000:0.0:1787826424.086681:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585282432 00000100:00100000:0.0:1787826424.086724:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 549 00000100:00100000:0.0:1787826424.086731:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff97689916f850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585282432:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826424.086746:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab15d1 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83325 expref: -99 pid: 55695 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826424.086802:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826424.086806:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff97689916f850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585282432:12345-192.168.203.139@tcp:104: Request processed in 77us (270us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826424.086811:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 549 00000100:00100000:0.0:1787826424.094970:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826424.095063:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585282560 00000100:00100000:1.0:1787826424.095073:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585282560 00000100:00100000:1.0:1787826424.095130:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 550 00000100:00100000:1.0:1787826424.095140:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aec780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585282560:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826424.095158:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab15d8 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83333 expref: -99 pid: 55695 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826424.095241:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826424.095248:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aec780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585282560:12345-192.168.203.139@tcp:104: Request processed in 110us (287us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826424.095256:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 550 00010000:00010000:3.0:1787826424.095380:0:49796:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826424.095387:0:49796:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab15d8 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83333 expref: -99 pid: 55695 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.095396:0:49796:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515400 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826424.095401:0:49796:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab15d8 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x3d:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83333 expref: -99 pid: 55695 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.095409:0:49796:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab15d8 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x3d:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83333 expref: -99 pid: 55695 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.095418:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab15d8 lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x3d:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b83333 expref: -99 pid: 55695 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826424.095443:0:49796:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x3d:0x0]: losing layout lock 00010000:00010000:3.0:1787826424.095486:0:49796:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab15d8 lrc: 2/0,0 mode: --/PR res: [0x200000405:0x3d:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83333 expref: -99 pid: 55695 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.095494:0:49796:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826424.095511:0:49796:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab15d8 lrc: 1/0,0 mode: --/PR res: [0x200000405:0x3d:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83333 expref: -99 pid: 55695 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.095531:0:49796:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab15d8 lrc: 0/0,0 mode: --/PR res: [0x200000405:0x3d:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83333 expref: -99 pid: 55695 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.095550:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.095552:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.095554:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826424.095562:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826424.095577:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563974144:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826424.103431:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563974144:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826424.108076:0:55695:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55695:1874670563974272:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826424.108110:0:55695:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab15e6 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83364 expref: -99 pid: 55695 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.108127:0:55695:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab15e6 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83364 expref: -99 pid: 55695 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826424.108142:0:55695:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff976889ef9a40 x1874670563974272/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 376/480 e 0 to 0 dl 1787826440 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826424.108153:0:55695:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab15e6 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83364 expref: -99 pid: 55695 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826424.108165:0:55695:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab15e6 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83364 expref: -99 pid: 55695 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.108178:0:55695:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826424.108186:0:55695:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1787826424.108215:0:55695:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab15e6 00000080:00010000:3.0:1787826424.108224:0:55695:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab15e6 from it ffffb2d442213bc8 00010000:00010000:3.0:1787826424.108228:0:55695:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab15e6 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83364 expref: -99 pid: 55695 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.108239:0:55695:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab15e6 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83364 expref: -99 pid: 55695 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.108265:0:55695:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab15e6 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83364 expref: -99 pid: 55695 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.108277:0:55695:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab15e6 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83364 expref: -99 pid: 55695 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1787826424.108286:0:55695:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab15e6 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83364 expref: -99 pid: 55695 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.108323:0:55695:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab15e6 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83364 expref: -99 pid: 55695 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.108332:0:55695:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab15e6 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83364 expref: -99 pid: 55695 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826424.108349:0:55695:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826424.108359:0:55695:0:(lov_io.c:693:lov_io_mirror_init()) [0x200000405:0x1d:0x0]: flr state: 1, move mirror from 1 to 1, have retried: 0, mirror count: 2 00020000:00200000:3.0:1787826424.108363:0:55695:0:(lov_io.c:750:lov_io_mirror_init()) use delayed RPC state for this IO 00020000:00200000:3.0:1787826424.108383:0:55695:0:(lov_io.c:1253:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:3.0:1787826424.108412:0:55695:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:3.0:1787826424.108424:0:55695:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab15e6 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83364 expref: -99 pid: 55695 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826424.108432:0:55695:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab15e6 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83364 expref: -99 pid: 55695 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:3.0:1787826424.108439:0:55695:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab15e6 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83364 expref: -99 pid: 55695 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826424.108453:0:55695:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826424.108458:0:55695:0:(lov_io.c:693:lov_io_mirror_init()) [0x200000405:0x1d:0x0]: flr state: 1, move mirror from 1 to 1, have retried: 0, mirror count: 2 00020000:00200000:3.0:1787826424.108460:0:55695:0:(lov_io.c:750:lov_io_mirror_init()) use delayed RPC state for this IO 00020000:00200000:3.0:1787826424.108462:0:55695:0:(lov_io.c:1253:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:3.0:1787826424.108470:0:55695:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826424.108697:0:55695:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 4294967294 -> 33 00010000:00010000:3.0:1787826424.108703:0:55695:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab15e6 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83364 expref: -99 pid: 55695 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826424.108712:0:55695:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab15e6 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83364 expref: -99 pid: 55695 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:3.0:1787826424.108745:0:55695:0:(file.c:479:ll_file_release()) START file . :VOLATILE:0000:5E960C5:fd=03:[0x200000405:0x3d:0x0](ffff976890b380c8), flags 554002 00000002:00100000:3.0:1787826424.108785:0:55695:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889ef8780 x1874670563973376/t8589935181(8589935181) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826440 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826424.108796:0:55695:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1787826424.108802:0:55695:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff97688cb5d680 x1874670563974528/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826424.108899:0:55695:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55695:1874670563974528:192.168.203.139@tcp:35:lfs.0 00000100:00100000:3.0:1787826424.108982:0:55695:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25900 going to sleep for 11 seconds 00000100:00100000:3.0:1787826424.113182:0:55695:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55695:1874670563974528:192.168.203.139@tcp:35:lfs.0 00000002:00100000:3.0:1787826424.113230:0:55695:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889ef8780 x1874670563973376/t8589935181(8589935181) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826440 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1787826424.113289:0:55695:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154002 00000002:00100000:3.0:1787826424.113342:0:55695:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889efa1c0 x1874670563971840/t8589935177(8589935177) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826440 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826424.113353:0:55695:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1787826424.113358:0:55695:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff97688cb5c000 x1874670563974784/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826424.113398:0:55695:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55695:1874670563974784:192.168.203.139@tcp:35:lfs.0 00000100:00100000:3.0:1787826424.113474:0:55695:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25900 going to sleep for 11 seconds 00000100:00100000:3.0:1787826424.116994:0:55695:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55695:1874670563974784:192.168.203.139@tcp:35:lfs.0 00000002:00100000:3.0:1787826424.117029:0:55695:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889efa1c0 x1874670563971840/t8589935177(8589935177) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826440 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826424.141715:0:55697:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826424.141748:0:55697:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826424.141767:0:55697:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826424.141782:0:55697:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826424.141790:0:55697:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826424.141797:0:55697:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826424.141801:0:55697:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826424.141808:0:55697:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d442233b58 00010000:00010000:0.0:1787826424.141812:0:55697:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826424.141820:0:55697:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826424.141895:0:55697:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826424.141903:0:55697:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=24 00000080:00200000:0.0:1787826424.141909:0:55697:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826424.141919:0:55697:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), flags=8 00000080:00200000:0.0:1787826424.141923:0:55697:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=getattr 00000002:00010000:0.0:1787826424.141937:0:55697:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f60b.sanity-flr,[0x0:0x0:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787826424.141981:0:55697:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55697 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.141990:0:55697:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55697 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.141999:0:55697:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55697 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826424.142022:0:55697:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeda40 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55697:1874670563974912:192.168.203.139@tcp:101:getfattr.0 00000100:00100000:0.0:1787826424.143082:0:55697:0:(client.c:2715:ptlrpc_set_wait()) set ffff976891c9a000 going to sleep for 11 seconds 00000100:00100000:0.0:1787826424.152769:0:55697:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeda40 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55697:1874670563974912:192.168.203.139@tcp:101:getfattr.0 00010000:00010000:0.0:1787826424.152796:0:55697:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 4/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83387 expref: -99 pid: 55697 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.152820:0:55697:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83387 expref: -99 pid: 55697 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.152879:0:55697:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83387 expref: -99 pid: 55697 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.152891:0:55697:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83387 expref: -99 pid: 55697 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.152901:0:55697:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83387 expref: -99 pid: 55697 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.152915:0:55697:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83387 expref: -99 pid: 55697 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.152923:0:55697:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83387 expref: -99 pid: 55697 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.152930:0:55697:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83387 expref: -99 pid: 55697 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826424.152941:0:55697:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff976882aeda40 x1874670563974912/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 584/720 e 0 to 0 dl 1787826440 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'getfattr.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826424.152953:0:55697:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83387 expref: -99 pid: 55697 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826424.152967:0:55697:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83387 expref: -99 pid: 55697 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.152978:0:55697:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/29 (0 0) 00000080:00200000:0.0:1787826424.152985:0:55697:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826424.153030:0:55697:0:(namei.c:147:ll_iget()) got inode: [0x200000405:0x1d:0x0](ffff9768901a80c8): rc = 0 00000080:00200000:0.0:1787826424.153044:0:55697:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 33 -> 33 00000080:00010000:0.0:1787826424.153050:0:55697:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab15ed 00000080:00010000:0.0:1787826424.153060:0:55697:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:0.0:1787826424.153063:0:55697:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab15ed from it ffffb2d442233b20 00010000:00010000:0.0:1787826424.153067:0:55697:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83387 expref: -99 pid: 55697 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.153077:0:55697:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83387 expref: -99 pid: 55697 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1787826424.153113:0:55697:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826424.153119:0:55697:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:0.0:1787826424.153144:0:55697:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826424.153153:0:55697:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83387 expref: -99 pid: 55697 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826424.153163:0:55697:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83387 expref: -99 pid: 55697 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826424.153170:0:55697:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826424.153175:0:55697:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:0.0:1787826424.153179:0:55697:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab15ed 00000080:00010000:0.0:1787826424.153183:0:55697:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab15ed from it ffffb2d442233cb8 00010000:00010000:0.0:1787826424.153186:0:55697:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83387 expref: -99 pid: 55697 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826424.153206:0:55697:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83387 expref: -99 pid: 55697 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826424.153230:0:55697:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 31 restore needed 0 00000001:00010000:0.0:1787826424.153240:0:55697:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:0.0:1787826424.153258:0:55697:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826424.153261:0:55697:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e55330, osc lock ffff9768849a2618, flags 0x400001002 00010000:00010000:0.0:1787826424.153278:0:55697:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826424.153289:0:55697:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826424.153297:0:55697:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826424.153304:0:55697:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826424.153340:0:55697:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab15f4 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55697 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.153353:0:55697:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab15f4 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55697 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826424.153398:0:55697:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1787826424.153411:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1787826424.153428:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563975040:192.168.203.139@tcp:101:getfattr.0 00000100:00100000:1.0:1787826424.160977:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826424.161057:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585283328 00000100:00100000:0.0:1787826424.161066:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585283328 00000100:00100000:0.0:1787826424.161118:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 551 00000100:00100000:0.0:1787826424.161128:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0163840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:8435:x1874670585283328:12345-192.168.203.139@tcp:106: 00010000:00010000:0.0:1787826424.161143:0:42210:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826424.161156:0:42210:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:0.0:1787826424.161184:0:42210:0:(cl_object.c:322:cl_object_glimpse()) header@000000001b7ef6a2[0x110, 1, [0x200000405:0x3c:0x0] hash] 00000020:00010000:0.0:1787826424.161186:0:42210:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826423 blocks: 0 00000020:00010000:0.0:1787826424.161202:0:42210:0:(cl_object.c:322:cl_object_glimpse()) header@000000006183ef1a[0x116, 2, [0x2c0000401:0x10d:0x0]] 00000020:00010000:0.0:1787826424.161204:0:42210:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826423 blocks: 0 00000100:00100000:0.0:1787826424.161257:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0163840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:8435:x1874670585283328:12345-192.168.203.139@tcp:106: Request processed in 130us (290us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826424.161264:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 551 00010000:00010000:0.0:1787826424.163889:0:2402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab15f4 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55697 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.163918:0:2402:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab15f4 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55697 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.163931:0:2402:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab15f4 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55697 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.163940:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab15f4 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55697 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.163948:0:2402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab15f4 lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55697 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826424.163961:0:2402:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:0.0:1787826424.163968:0:2402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1787826424.163994:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab15f4 lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55697 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.164003:0:2402:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab15f4 lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55697 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.164010:0:2402:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab15f4 lrc: 4/0,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55697 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.164018:0:2402:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925cc00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:0.0:1787826424.164024:0:2402:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab15f4 lrc: 5/0,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55697 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.164036:0:2402:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787826424.164042:0:2402:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab15f4 lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55697 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.164050:0:2402:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab15f4 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55697 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826424.164070:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563975040:192.168.203.139@tcp:101:getfattr.0 00000100:00100000:0.0:1787826424.189106:0:55701:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef83c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55701:1874670563975168:192.168.203.139@tcp:34:lfs.0 00000080:00200008:0.0:1787826424.189341:0:55701:0:(file.c:479:ll_file_release()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00010000:00010000:0.0:1787826424.189372:0:55701:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9600/0x8e7c99ee62ab0c1c lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b818c9 expref: -99 pid: 55483 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000100:00100000:1.0:1787826424.221935:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826424.222076:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585283456 00000100:00100000:0.0:1787826424.222085:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585283456 00000100:00100000:0.0:1787826424.222215:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 552 00000100:00100000:0.0:1787826424.222226:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0160780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:30696:x1874670585283456:12345-192.168.203.139@tcp:106: 00010000:00010000:0.0:1787826424.222243:0:42211:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826424.222255:0:42211:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00010000:0.0:1787826424.222287:0:42211:0:(cl_object.c:322:cl_object_glimpse()) header@000000001b7ef6a2[0x110, 1, [0x200000405:0x3c:0x0] hash] 00000020:00010000:0.0:1787826424.222290:0:42211:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826423 blocks: 0 00000020:00010000:0.0:1787826424.222294:0:42211:0:(cl_object.c:322:cl_object_glimpse()) header@000000006183ef1a[0x116, 2, [0x2c0000401:0x10d:0x0]] 00000020:00010000:0.0:1787826424.222296:0:42211:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826423 blocks: 0 00000100:00100000:0.0:1787826424.222349:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0160780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:30696:x1874670585283456:12345-192.168.203.139@tcp:106: Request processed in 124us (422us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826424.222356:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 552 00000080:00200000:0.0:1787826424.228627:0:55703:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787826424.228634:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf18b0 (ffff976890d10368/0) at cl_lock_release():245 00000020:00010000:0.0:1787826424.228637:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf18b0 (ffff976890d10368/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826424.228642:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757ccb0 (ffff976890d10f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826424.228646:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf18b0 (ffff976890d10368/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826424.228649:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757ccb0 (ffff976890d10368/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826424.228661:0:55703:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 33 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787826424.228713:0:55703:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154000 00010000:00010000:0.0:1787826424.228738:0:55703:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab15fb lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b833b1 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826424.228880:0:55703:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826424.228900:0:55703:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826424.228911:0:55703:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826424.228920:0:55703:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826424.228933:0:55703:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826424.228939:0:55703:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826424.228943:0:55703:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826424.228947:0:55703:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44222bbf8 00010000:00010000:0.0:1787826424.228951:0:55703:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826424.228958:0:55703:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826424.228972:0:55703:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826424.228978:0:55703:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:0.0:1787826424.228983:0:55703:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826424.228987:0:55703:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:0.0:1787826424.228991:0:55703:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 46 00000080:00200008:0.0:1787826424.228998:0:55703:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 1154402 00000002:00010000:0.0:1787826424.229011:0:55703:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114403 00010000:00010000:0.0:1787826424.229120:0:55703:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1609 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 55703 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.229128:0:55703:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1609 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55703 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826424.229138:0:55703:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826424.229147:0:55703:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aeda40 x1874670563975808/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826424.229156:0:55703:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1609 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55703 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826424.229178:0:55703:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeda40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55703:1874670563975808:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826424.231950:0:55703:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e540 going to sleep for 11 seconds 00000100:00100000:1.0:1787826424.241961:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826424.244711:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585283584 00000100:00100000:0.0:1787826424.244725:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585283584 00000100:00100000:0.0:1787826424.244856:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 553 00000100:00100000:0.0:1787826424.244867:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0160000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:9966:x1874670585283584:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826424.244885:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83387 expref: -99 pid: 55697 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826424.244967:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826424.244974:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0160000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:9966:x1874670585283584:12345-192.168.203.139@tcp:104: Request processed in 109us (3022us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826424.244982:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 553 00010000:00010000:3.0:1787826424.245067:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826424.245075:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83387 expref: -99 pid: 55697 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826424.245083:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925c600 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826424.245089:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83387 expref: -99 pid: 55697 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826424.245103:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83387 expref: -99 pid: 55697 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826424.245112:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b83387 expref: -99 pid: 55697 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826424.245137:0:42220:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab15fb lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b833b1 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826424.245177:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 2/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83387 expref: -99 pid: 55697 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826424.245183:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826424.245219:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 1/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83387 expref: -99 pid: 55697 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826424.245228:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab15ed lrc: 0/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83387 expref: -99 pid: 55697 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826424.245243:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.245245:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.245247:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826424.245256:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826424.245270:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563975936:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826424.249009:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563975936:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826424.266024:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826424.266118:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585283712 00000100:00100000:0.0:1787826424.266128:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585283712 00000100:00100000:0.0:1787826424.266183:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 554 00000100:00100000:0.0:1787826424.266202:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0162940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:16564:x1874670585283712:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826424.266221:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1610 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b833e2 expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826424.266302:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826424.266308:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0162940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:16564:x1874670585283712:12345-192.168.203.139@tcp:104: Request processed in 117us (295us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826424.266315:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 554 00010000:00010000:3.0:1787826424.266404:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826424.266415:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1610 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b833e2 expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.266424:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514400 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826424.266429:0:42221:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1610 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x3e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b833e2 expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.266439:0:42221:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1610 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x3e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b833e2 expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.266448:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1610 lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x3e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b833e2 expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826424.266473:0:42221:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x3e:0x0]: losing layout lock 00010000:00010000:3.0:1787826424.266515:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1610 lrc: 2/0,0 mode: --/PR res: [0x200000405:0x3e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b833e2 expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.266522:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826424.266556:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1610 lrc: 1/0,0 mode: --/PR res: [0x200000405:0x3e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b833e2 expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.266564:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1610 lrc: 0/0,0 mode: --/PR res: [0x200000405:0x3e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b833e2 expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.266577:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.266579:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.266581:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826424.266589:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787826424.266603:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563976576:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826424.277689:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563976576:192.168.203.139@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1787826424.287411:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826424.287606:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826424.288273:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585283840 00000100:00100000:0.0:1787826424.288283:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585283840 00000100:00100000:0.0:1787826424.288333:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 555 00000100:00100000:0.0:1787826424.288343:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0161a40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6518:x1874670585283840:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826424.288361:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1609 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x22153cd396b833cd expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826424.288451:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826424.288457:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0161a40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6518:x1874670585283840:12345-192.168.203.139@tcp:104: Request processed in 116us (858us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826424.288465:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 555 00010000:00010000:3.0:1787826424.288667:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826424.288674:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1609 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x22153cd396b833cd expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.288683:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925c800 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826424.288689:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1609 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x22153cd396b833cd expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.288697:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1609 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x22153cd396b833cd expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.288705:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1609 lrc: 3/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x22153cd396b833cd expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.288733:0:42220:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab15fb lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b833b1 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826424.288778:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1609 lrc: 2/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x22153cd396b833cd expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.288785:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826424.288811:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1609 lrc: 1/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x22153cd396b833cd expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.288819:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab1609 lrc: 0/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x22153cd396b833cd expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.288858:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.288860:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.288861:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826424.288869:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826424.288883:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563977088:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826424.295230:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563977088:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826424.302236:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826424.302333:0:42210:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:0.0:1787826424.302352:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585283968 00000100:00100000:0.0:1787826424.302361:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585283968 00000100:00100000:0.0:1787826424.302427:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 556 00000100:00100000:0.0:1787826424.302437:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768895b8850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30694:x1874670585283968:12345-192.168.203.139@tcp:106: 00010000:00010000:0.0:1787826424.302461:0:42210:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826424.302475:0:42210:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00000020:00010000:0.0:1787826424.302502:0:42210:0:(cl_object.c:322:cl_object_glimpse()) header@000000001b7ef6a2[0x110, 1, [0x200000405:0x3c:0x0] hash] 00000020:00010000:0.0:1787826424.302504:0:42210:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826423 blocks: 0 00000020:00010000:0.0:1787826424.302508:0:42210:0:(cl_object.c:322:cl_object_glimpse()) header@000000006183ef1a[0x116, 2, [0x2c0000401:0x10d:0x0]] 00000020:00010000:0.0:1787826424.302510:0:42210:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826423 blocks: 0 00000100:00100000:0.0:1787826424.302570:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768895b8850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30694:x1874670585283968:12345-192.168.203.139@tcp:106: Request processed in 134us (342us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826424.302580:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 556 00000080:00200000:0.0:1787826424.307513:0:55703:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787826424.307519:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf1da0 (ffff976890d10e18/0) at cl_lock_release():245 00000020:00010000:0.0:1787826424.307523:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf1da0 (ffff976890d10e18/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826424.307527:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757c830 (ffff976890d10ea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826424.307531:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf1da0 (ffff976890d10e18/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826424.307534:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757c830 (ffff976890d10e18/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826424.307545:0:55703:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 33 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787826424.307568:0:55703:0:(file.c:6313:ll_getattr_dentry()) START file . :VOLATILE:0000:29DC0D6:fd=03:[0x200000405:0x3e:0x0](ffff976890b3aa08), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826424.307575:0:55703:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x3e:0x0](ffff976890b3aa08),name=. :VOLATILE:0000:29DC0D6:fd=03 00000002:00010000:0.0:1787826424.307593:0:55703:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x3e:0x0]) in obj [0x200000405:0x3e:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826424.307643:0:55703:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab162c lrc: 3/1,0 mode: --/CR res: [0x200000405:0x3e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55703 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.307652:0:55703:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab162c lrc: 3/1,0 mode: --/CR res: [0x200000405:0x3e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55703 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.307661:0:55703:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab162c lrc: 3/1,0 mode: --/CR res: [0x200000405:0x3e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55703 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826424.307680:0:55703:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55703:1874670563977344:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826424.307757:0:55703:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e6c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826424.323123:0:55703:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55703:1874670563977344:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826424.323149:0:55703:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab162c lrc: 4/1,0 mode: --/CR res: [0x200000405:0x3e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83421 expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.323161:0:55703:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab162c lrc: 4/1,0 mode: --/PR res: [0x200000405:0x3e:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83421 expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.323173:0:55703:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab162c lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x3e:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83421 expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.323184:0:55703:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab162c lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x3e:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83421 expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.323205:0:55703:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab162c lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x3e:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83421 expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.323214:0:55703:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab162c lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x3e:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83421 expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.323222:0:55703:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab162c lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x3e:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83421 expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826424.323232:0:55703:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff976882aec780 x1874670563977344/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 576/720 e 0 to 0 dl 1787826440 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826424.323247:0:55703:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab162c lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x3e:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83421 expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787826424.323428:0:55703:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab162c lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x3e:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83421 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826424.323444:0:55703:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/62 (0 0) 00000080:00200000:1.0:1787826424.323461:0:55703:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000405:0x3e:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1787826424.323519:0:55703:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x3e:0x0]: layout version change: 4294967294 -> 1 00000080:00010000:1.0:1787826424.323525:0:55703:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x3e:0x0](ffff976890b3aa08) for lock 0x8e7c99ee62ab162c 00000080:00010000:1.0:1787826424.323530:0:55703:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x3e:0x0](ffff976890b3aa08) 00000080:00010000:1.0:1787826424.323533:0:55703:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab162c from it ffffb2d44222bcd8 00010000:00010000:1.0:1787826424.323648:0:55703:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab162c lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x3e:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83421 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826424.323658:0:55703:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c800/0x8e7c99ee62ab162c lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x3e:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83421 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787826424.323690:0:55703:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x3e:0x0] ignore/verify layout 0/1, layout version 27 restore needed 0 00000001:00010000:1.0:1787826424.323713:0:55703:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x3e:0x0] 00000008:00010000:1.0:1787826424.323731:0:55703:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1787826424.323734:0:55703:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e79230, osc lock ffff976883ff8ea0, flags 0x400001002 00010000:00010000:1.0:1787826424.323790:0:55703:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000026ddde58/0x8e7c99ee62ab1633 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826424.323802:0:55703:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000026ddde58/0x8e7c99ee62ab1633 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826424.324012:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563977472:192.168.203.139@tcp:101:lfs.0 00000020:00200000:1.0:1787826424.324147:0:55703:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1787826424.334244:0:2402:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9768877a4800: committing for last_committed 21474837022 gen 1 00010000:00010000:0.0:1787826424.334298:0:2402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000026ddde58/0x8e7c99ee62ab1633 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.334324:0:2402:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000026ddde58/0x8e7c99ee62ab1633 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83428 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.334338:0:2402:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000026ddde58/0x8e7c99ee62ab1633 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83428 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.334349:0:2402:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000026ddde58/0x8e7c99ee62ab1633 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83428 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826424.334363:0:2402:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000026ddde58/0x8e7c99ee62ab1633 lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83428 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826424.334374:0:2402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1787826424.334389:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000026ddde58/0x8e7c99ee62ab1633 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83428 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.334397:0:2402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000026ddde58/0x8e7c99ee62ab1633 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83428 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.334404:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000026ddde58/0x8e7c99ee62ab1633 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83428 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.334413:0:2402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000026ddde58/0x8e7c99ee62ab1633 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83428 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1787826424.334439:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0160b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563977472:192.168.203.139@tcp:101:lfs.0 00000080:00200000:1.0:1787826424.334669:0:55703:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x3e:0x0] updating i_size 0 i_blocks 0 00000020:00010000:1.0:1787826424.334676:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976890e938c8 (ffff97688386c998/0) at cl_lock_release():245 00000020:00010000:1.0:1787826424.334679:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976890e938c8 (ffff97688386c998/0) at cl_lock_cancel():142 00000020:00010000:1.0:1787826424.334684:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e79230 (ffff976890d10908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1787826424.334688:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976890e938c8 (ffff97688386c998/0) at cl_lock_fini():72 00000020:00010000:1.0:1787826424.334691:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e79230 (ffff97688386c998/1) at cl_lock_fini():72 00000080:00200000:1.0:1787826424.334716:0:55703:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x3e:0x0] ignore/verify layout 0/1, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200080:1.0:1787826424.334739:0:55703:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=7ffd7610e5d4 00000002:00010000:1.0:1787826424.334765:0:55703:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 01004040000001 00010000:00010000:1.0:1787826424.334813:0:55703:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfa00/0x8e7c99ee62ab163a lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 55703 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826424.334821:0:55703:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x81000000001000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfa00/0x8e7c99ee62ab163a lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55703 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1787826424.334868:0:55703:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1787826424.334876:0:55703:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b0163480 x1874670563977728/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1787826424.334884:0:55703:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfa00/0x8e7c99ee62ab163a lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x81000000000000 pid: 55703 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826424.334902:0:55703:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0163480 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55703:1874670563977728:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826424.335061:0:55703:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3d80 going to sleep for 11 seconds 00000100:00100000:3.0:1787826424.339672:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826424.339688:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563977856:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826424.346145:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826424.348299:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585284096 00000100:00100000:0.0:1787826424.348312:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585284096 00000100:00100000:0.0:1787826424.348371:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 557 00000100:00100000:0.0:1787826424.348381:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0160b40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:30398:x1874670585284096:12345-192.168.203.139@tcp:106: 00010000:00010000:0.0:1787826424.348396:0:42211:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826424.348411:0:42211:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00000020:00010000:0.0:1787826424.348443:0:42211:0:(cl_object.c:322:cl_object_glimpse()) header@000000001b7ef6a2[0x110, 1, [0x200000405:0x3c:0x0] hash] 00000020:00010000:0.0:1787826424.348445:0:42211:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826423 blocks: 0 00000020:00010000:0.0:1787826424.348451:0:42211:0:(cl_object.c:322:cl_object_glimpse()) header@000000006183ef1a[0x116, 2, [0x2c0000401:0x10d:0x0]] 00000020:00010000:0.0:1787826424.348453:0:42211:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826423 blocks: 0 00000100:00100000:0.0:1787826424.348512:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0160b40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:30398:x1874670585284096:12345-192.168.203.139@tcp:106: Request processed in 132us (2376us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826424.348519:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 557 00010000:00010000:3.0:1787826424.350625:0:2403:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1641 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.350646:0:2403:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1641 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.350658:0:2403:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1641 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.350666:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1641 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.350674:0:2403:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1641 lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826424.350688:0:2403:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:3.0:1787826424.350697:0:2403:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826424.350724:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1641 lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.350733:0:2403:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1641 lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.350741:0:2403:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1641 lrc: 4/0,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.350747:0:2403:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514800 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826424.350753:0:2403:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1641 lrc: 5/0,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.350765:0:2403:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826424.350771:0:2403:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1641 lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.350779:0:2403:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1641 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826424.350797:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563977856:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826424.351524:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787826424.351538:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563977984:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826424.361742:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826424.361867:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585284224 00000100:00100000:0.0:1787826424.361878:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585284224 00000100:00100000:0.0:1787826424.361923:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 558 00000100:00100000:0.0:1787826424.361933:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b01621c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8545:x1874670585284224:12345-192.168.203.139@tcp:106: 00010000:00010000:0.0:1787826424.361948:0:53061:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826424.361967:0:53061:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00000020:00010000:0.0:1787826424.361997:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@000000001b7ef6a2[0x110, 1, [0x200000405:0x3c:0x0] hash] 00000020:00010000:0.0:1787826424.361999:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826423 blocks: 0 00000020:00010000:0.0:1787826424.362003:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@000000006183ef1a[0x116, 2, [0x2c0000401:0x10d:0x0]] 00000020:00010000:0.0:1787826424.362004:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826423 blocks: 0 00000100:00100000:0.0:1787826424.362212:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b01621c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8545:x1874670585284224:12345-192.168.203.139@tcp:106: Request processed in 280us (480us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826424.362221:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 558 00010000:00010000:3.0:1787826424.374292:0:2404:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab1648 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.374323:0:2404:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab1648 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.374336:0:2404:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab1648 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.374344:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab1648 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.374351:0:2404:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab1648 lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826424.374366:0:2404:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:3.0:1787826424.374372:0:2404:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826424.374394:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab1648 lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.374403:0:2404:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab1648 lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.374411:0:2404:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab1648 lrc: 4/0,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.374418:0:2404:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514e00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826424.374424:0:2404:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab1648 lrc: 5/0,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.374435:0:2404:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826424.374442:0:2404:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab1648 lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.374449:0:2404:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab1648 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826424.374468:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563977984:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826424.374794:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563978112:192.168.203.139@tcp:1:lfs.0 00000100:00100000:1.0:1787826424.384494:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826424.384588:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585284352 00000100:00100000:0.0:1787826424.384598:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585284352 00000100:00100000:0.0:1787826424.384654:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 559 00000100:00100000:0.0:1787826424.384665:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0163840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:14864:x1874670585284352:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826424.384683:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826424.384757:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826424.384764:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0163840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:14864:x1874670585284352:12345-192.168.203.139@tcp:104: Request processed in 101us (281us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826424.384772:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 559 00010000:00010000:3.0:1787826424.384837:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826424.384845:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826424.384853:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925dc00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826424.384859:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400020000 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1787826424.384881:0:42221:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x3c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787826424.384894:0:42221:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x3c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787826424.384935:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 2/0,0 mode: --/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826424.384948:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826424.384979:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 1/0,0 mode: --/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826424.384988:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000a0dc14ee/0x8e7c99ee62ab15a0 lrc: 0/0,0 mode: --/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x22153cd396b8327d expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826424.385010:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.385012:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.385014:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826424.385027:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787826424.385040:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563978240:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826424.395249:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563978240:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826424.400903:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563978112:192.168.203.139@tcp:1:lfs.0 00000080:00200000:1.0:1787826424.415272:0:55703:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:1.0:1787826424.415279:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf09e0 (ffff976890d103f8/0) at cl_lock_release():245 00000020:00010000:1.0:1787826424.415282:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf09e0 (ffff976890d103f8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1787826424.415288:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757ceb0 (ffff976890d10a28/1) at cl_lock_cancel():142 00000020:00010000:1.0:1787826424.415292:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf09e0 (ffff976890d103f8/0) at cl_lock_fini():72 00000020:00010000:1.0:1787826424.415295:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757ceb0 (ffff976890d103f8/1) at cl_lock_fini():72 00000080:00200000:1.0:1787826424.415330:0:55703:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 33 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1787826424.477543:0:55703:0:(file.c:5224:ll_file_seek()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0], offset: 0, type: (null) 00000080:00200000:1.0:1787826424.477561:0:55703:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 1 restore needed 0 00000001:00010000:1.0:1787826424.477571:0:55703:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:1.0:1787826424.477595:0:55703:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1787826424.477599:0:55703:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e79230, osc lock ffff976883ff8ea0, flags 0x400001002 00010000:00010000:1.0:1787826424.477623:0:55703:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab164f lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83459 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826424.477642:0:55703:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab164f lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83459 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:1.0:1787826424.477656:0:55703:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1787826424.477661:0:55703:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab164f lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83459 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826424.477671:0:55703:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab164f lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83459 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:1.0:1787826424.477685:0:55703:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1787826424.477695:0:55703:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:1.0:1787826424.477699:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976890e938c8 (ffff97688386c998/0) at cl_lock_release():245 00000020:00010000:1.0:1787826424.477702:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976890e938c8 (ffff97688386c998/0) at cl_lock_cancel():142 00000020:00010000:1.0:1787826424.477705:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e79230 (ffff976890d10908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1787826424.477709:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976890e938c8 (ffff97688386c998/0) at cl_lock_fini():72 00000020:00010000:1.0:1787826424.477712:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e79230 (ffff97688386c998/1) at cl_lock_fini():72 00000080:00200000:1.0:1787826424.477725:0:55703:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 33 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1787826424.477733:0:55703:0:(file.c:5262:ll_file_seek()) COMPLETED file f60b.sanity-flr:[0x200000405:0x1d:0x0], offset: 0, type: (null), rc = 0 00000080:00200008:1.0:1787826424.477738:0:55703:0:(file.c:5224:ll_file_seek()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0], offset: 0, type: SEEK_HOLE 00000080:00200000:1.0:1787826424.477742:0:55703:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 33 restore needed 0 00020000:00200000:1.0:1787826424.477749:0:55703:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1787826424.477766:0:55703:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000080:00200000:1.0:1787826424.477773:0:55703:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1787826424.477779:0:55703:0:(file.c:1848:ll_io_set_mirror()) f60b.sanity-flr: desiginated mirror: 0 00000080:00200000:1.0:1787826424.477781:0:55703:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:1.0:1787826424.477786:0:55703:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1787826424.477790:0:55703:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 18446744073709551615] 00000008:00010000:1.0:1787826424.477797:0:55703:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 8 00000008:00010000:1.0:1787826424.477799:0:55703:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e79230, osc lock ffff976883ff8ea0, flags 0x0 00010000:00010000:1.0:1787826424.477808:0:55703:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab164f lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83459 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826424.477818:0:55703:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab164f lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83459 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826424.477858:0:55703:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab164f lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83459 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826424.477867:0:55703:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab164f lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83459 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826424.477874:0:55703:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab164f lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83459 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787826424.477885:0:55703:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:1.0:1787826424.477889:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885e73998 (ffff97688386c998/0) at cl_lock_release():245 00000020:00010000:1.0:1787826424.477892:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885e73998 (ffff97688386c998/0) at cl_lock_cancel():142 00000020:00010000:1.0:1787826424.477894:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e79230 (ffff976890d10908/1) at cl_lock_cancel():142 00010000:00010000:1.0:1787826424.477898:0:55703:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab164f lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83459 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826424.477906:0:55703:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab164f lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83459 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00010000:1.0:1787826424.477915:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885e73998 (ffff97688386c998/0) at cl_lock_fini():72 00000020:00010000:1.0:1787826424.477917:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e79230 (ffff97688386c998/1) at cl_lock_fini():72 00000080:00200000:1.0:1787826424.477926:0:55703:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 33 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1787826424.477930:0:55703:0:(file.c:5262:ll_file_seek()) COMPLETED file f60b.sanity-flr:[0x200000405:0x1d:0x0], offset: 0, type: SEEK_HOLE, rc = -6 00000080:00200080:1.0:1787826424.477964:0:55703:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=66f4 arg=559 00000080:00200008:1.0:1787826424.477979:0:55703:0:(file.c:2485:do_file_read_iter()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0], ppos: 0, count: 67108864 00000080:00200000:1.0:1787826424.477989:0:55703:0:(file.c:2090:ll_file_io_generic()) f60b.sanity-flr: read ppos: 0, bytes: 67108864 00000080:00200000:1.0:1787826424.477995:0:55703:0:(file.c:1848:ll_io_set_mirror()) f60b.sanity-flr: desiginated mirror: 0 00000020:00200000:1.0:1787826424.478001:0:55703:0:(cl_io.c:218:cl_io_rw_init()) header@00000000a0ca5d5e[0x110, 2, [0x200000405:0x1d:0x0] hash] 00000020:00200000:1.0:1787826424.478003:0:55703:0:(cl_io.c:218:cl_io_rw_init()) io range: 1 [0, 67108864) 2048 0 00000080:00200000:1.0:1787826424.478006:0:55703:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 33 restore needed 0 00000080:00200000:1.0:1787826424.478014:0:55703:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 16383] 00020000:00200000:1.0:1787826424.478021:0:55703:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 67108864) 67108864, 67108864 00020000:00200000:1.0:1787826424.478024:0:55703:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1787826424.478033:0:55703:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 16383] 00000008:00010000:1.0:1787826424.478038:0:55703:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 11 00000008:00010000:1.0:1787826424.478041:0:55703:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e79230, osc lock ffff976883ff8ea0, flags 0x40000 00000080:00200000:1.0:1787826424.478045:0:55703:0:(vvp_io.c:874:vvp_io_read_start()) f60b.sanity-flr: read [0, 67108864) 00000001:00010000:1.0:1787826424.478050:0:55703:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:1.0:1787826424.478054:0:55703:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1787826424.478056:0:55703:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e795b0, osc lock ffff976883ff8750, flags 0x400001002 00010000:00010000:1.0:1787826424.478064:0:55703:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab164f lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83459 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826424.478074:0:55703:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab164f lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83459 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00200000:1.0:1787826424.478083:0:55703:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1787826424.478085:0:55703:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab164f lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83459 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826424.478092:0:55703:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab164f lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83459 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00000020:00200000:1.0:1787826424.478101:0:55703:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1787826424.478105:0:55703:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:1.0:1787826424.478107:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976890e938c8 (ffff97688386c998/0) at cl_lock_release():245 00000020:00010000:1.0:1787826424.478110:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976890e938c8 (ffff97688386c998/0) at cl_lock_cancel():142 00000020:00010000:1.0:1787826424.478112:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e795b0 (ffff976890d10908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1787826424.478115:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976890e938c8 (ffff97688386c998/0) at cl_lock_fini():72 00000020:00010000:1.0:1787826424.478117:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e795b0 (ffff97688386c998/1) at cl_lock_fini():72 00000080:00200000:1.0:1787826424.478122:0:55703:0:(vvp_io.c:932:vvp_io_read_start()) count:67108864 iocb pos:0 00000080:00200000:1.0:1787826424.478128:0:55703:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), size=67108864 (max 335544320), offset=0=0x0, pages 16384 (max 81920), lockless, parallel 00000020:00010000:1.0:1787826424.478134:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885e73998 (ffff97688386c998/0) at cl_lock_release():245 00000020:00010000:1.0:1787826424.478136:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885e73998 (ffff97688386c998/0) at cl_lock_cancel():142 00000020:00010000:1.0:1787826424.478138:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e79230 (ffff976890d10908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1787826424.478140:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885e73998 (ffff97688386c998/0) at cl_lock_fini():72 00000020:00010000:1.0:1787826424.478142:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e79230 (ffff97688386c998/1) at cl_lock_fini():72 00000080:00200000:1.0:1787826424.478145:0:55703:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000020:00200000:1.0:1787826424.478149:0:55703:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1787826424.478151:0:55703:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 16383] 00000080:00200000:1.0:1787826424.478156:0:55703:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 33 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1787826424.478159:0:55703:0:(file.c:2240:ll_file_io_generic()) f60b.sanity-flr: 1 io complete with rc: 0, result: 0, restart: 0 00000020:00200000:1.0:1787826424.478161:0:55703:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1787826424.478228:0:55703:0:(file.c:2326:ll_file_io_generic()) iot: 1, result: 0 00000080:00200008:1.0:1787826424.478240:0:55703:0:(file.c:5401:ll_fsync()) START file: name=. :VOLATILE:0000:29DC0D6:fd=03, fid=[0x200000405:0x3e:0x0], start=0, end=9223372036854775807, datasync=0, need_sync 1/0 00000100:00100000:1.0:1787826424.478293:0:55703:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55703:1874670563978496:192.168.203.139@tcp:44:lfs.0 00000100:00100000:1.0:1787826424.479295:0:55703:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3240 going to sleep for 11 seconds 00000100:00100000:1.0:1787826424.493491:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826424.494134:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585284608 00000100:00100000:0.0:1787826424.494141:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585284608 00000100:00100000:0.0:1787826424.494180:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 560 00000100:00100000:0.0:1787826424.494198:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0160000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8436:x1874670585284608:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826424.494215:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab15a7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000030000 nid: local remote: 0x22153cd396b8328b expref: -99 pid: 55686 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826424.494267:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826424.494274:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0160000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8436:x1874670585284608:12345-192.168.203.139@tcp:104: Request processed in 87us (794us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826424.494281:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 560 00000080:00200000:1.0:1787826424.502071:0:55703:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x3e:0x0] updating i_size 0 i_blocks 0 00000080:00200000:1.0:1787826424.502125:0:55703:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x3e:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200080:0.0:1787826424.523607:0:55703:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=801066da arg=7ffd7610e4d0 00000080:00200000:0.0:1787826424.523632:0:55703:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 33 restore needed 0 00020000:00200000:0.0:1787826424.523645:0:55703:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:1.0:1787826424.523863:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670563978880:192.168.203.139@tcp:1:lfs.0 00000100:00100000:0.0:1787826424.532816:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670563978880:192.168.203.139@tcp:1:lfs.0 00000100:00100000:3.0:1787826424.544812:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:307626ff-a376-4faf-93e0-9f55b723d964:2403:1874670563979392:192.168.203.139@tcp:400:kworker.0 00000100:00100000:3.0:1787826424.546473:0:55703:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef92c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55703:1874670563979264:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826424.547332:0:55703:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab1656 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x3e:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83475 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.547351:0:55703:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab1656 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x3e:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83475 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826424.547385:0:55703:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff976889ef92c0 x1874670563979264/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 376/480 e 0 to 0 dl 1787826440 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826424.547400:0:55703:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab1656 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x3e:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83475 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826424.547653:0:55703:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab1656 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x3e:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83475 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.547669:0:55703:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/62 (0 0) 00000080:00200000:3.0:1787826424.547676:0:55703:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000405:0x3e:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1787826424.547701:0:55703:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x3e:0x0](ffff976890b3aa08) for lock 0x8e7c99ee62ab1656 00000080:00010000:3.0:1787826424.547711:0:55703:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1656 from it ffffb2d44222ba78 00010000:00010000:3.0:1787826424.547715:0:55703:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab1656 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x3e:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83475 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.547727:0:55703:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab1656 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x3e:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83475 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.547761:0:55703:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab1656 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x3e:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83475 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.547774:0:55703:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab1656 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x3e:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83475 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1787826424.547783:0:55703:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x3e:0x0](00000000ae00023d) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab1656 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x3e:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83475 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826424.548350:0:55703:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x3e:0x0]: layout version change: 4294967294 -> 1 00010000:00010000:3.0:1787826424.548376:0:55703:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab1656 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x3e:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83475 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.548405:0:55703:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab1656 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x3e:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83475 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:3.0:1787826424.548432:0:55703:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826424.548451:0:55703:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:3.0:1787826424.548469:0:55703:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1787826424.548473:0:55703:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff9768837ffcb0, osc lock ffff9768984efd40, flags 0x0 00010000:00010000:3.0:1787826424.548531:0:55703:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976889533000 type 11 mode 2 res 10737419265/269 (0 18446744073709551615) 00010000:00010000:3.0:1787826424.548574:0:55703:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.548584:0:55703:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.548595:0:55703:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 3/0,1 mode: --/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826424.548620:0:55703:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55703:1874670563979648:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826424.548904:0:55703:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25540 going to sleep for 11 seconds 00000100:00100000:1.0:1787826424.553494:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826424.553788:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585284736 00000100:00100000:0.0:1787826424.553801:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585284736 00000100:00100000:0.0:1787826424.553915:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 561 00000100:00100000:0.0:1787826424.553928:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0162940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:30696:x1874670585284736:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826424.553952:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000026ddde58/0x8e7c99ee62ab1633 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b83428 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826424.554053:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826424.554060:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0162940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:30696:x1874670585284736:12345-192.168.203.139@tcp:104: Request processed in 135us (583us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826424.554067:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 561 00010000:00010000:3.0:1787826424.554900:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826424.554913:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000026ddde58/0x8e7c99ee62ab1633 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b83428 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.554925:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976890fdee00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826424.554932:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000026ddde58/0x8e7c99ee62ab1633 lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x22153cd396b83428 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826424.554964:0:42220:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x3e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787826424.554981:0:42220:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x3e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787826424.555046:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000026ddde58/0x8e7c99ee62ab1633 lrc: 2/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b83428 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.555054:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826424.555092:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000026ddde58/0x8e7c99ee62ab1633 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b83428 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.555101:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000026ddde58/0x8e7c99ee62ab1633 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b83428 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.555121:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.555123:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.555125:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826424.557319:0:55703:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55703:1874670563979648:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826424.557358:0:55703:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 4/0,1 mode: --/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.557374:0:55703:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 4/0,1 mode: --/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.557383:0:55703:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 4/0,1 mode: --/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826424.557908:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826424.558108:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585284864 00000100:00100000:1.0:1787826424.558118:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585284864 00000100:00100000:1.0:1787826424.558178:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 562 00000100:00100000:1.0:1787826424.558198:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aed680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:10340:x1874670585284864:12345-192.168.203.139@tcp:105: 00010000:00010000:1.0:1787826424.558214:0:42210:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 5/0,1 mode: --/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826424.558226:0:42210:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 5/0,1 mode: --/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826424.558236:0:42210:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 5/0,1 mode: --/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826424.558246:0:42210:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826424.558262:0:42210:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 6/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826424.558272:0:42210:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 6/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826424.558374:0:42210:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff976888fb9800) 00000100:00100000:1.0:1787826424.558455:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aed680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:10340:x1874670585284864:12345-192.168.203.139@tcp:105: Request processed in 267us (551us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826424.558464:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 562 00010000:00010000:3.0:1787826424.559156:0:55703:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.559171:0:55703:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.559197:0:55703:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826424.559215:0:55703:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.559229:0:55703:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.559238:0:55703:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826424.560551:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563979776:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826424.560582:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826424.560593:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563979904:192.168.203.139@tcp:2:lfs.0 00000020:00010000:0.0:1787826424.564220:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976890fb7308 (ffff976890d10e18/0) at cl_lock_release():245 00000020:00010000:0.0:1787826424.564226:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976890fb7308 (ffff976890d10e18/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826424.564234:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768837ffcb0 (ffff976890d10c68/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787826424.564246:0:55703:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.564265:0:55703:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787826424.564285:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976890fb7308 (ffff976890d10e18/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826424.564288:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768837ffcb0 (ffff976890d10e18/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826424.564313:0:55703:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x3e:0x0] ignore/verify layout 0/0, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787826424.564332:0:55703:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file . :VOLATILE:0000:29DC0D6:fd=03:[0x200000405:0x3e:0x0](ffff976890b3aa08) current size 0, valid attrs 100101f0, mode 9768, uid 0, gid 0, new size 4294967296, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826424.000000000, rc 0 00000080:00200008:0.0:1787826424.564417:0:55703:0:(llite_lib.c:2427:ll_setattr_raw()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8) current size 0, valid attrs 101f0, mode 9768, uid 0, gid 0, new size 0, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826424.000000000 00010000:00010000:0.0:1787826424.564461:0:55703:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514600/0x8e7c99ee62ab161e lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b8340c expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826424.564491:0:55703:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab15fb lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b833b1 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00010000:00010000:0.0:1787826424.564552:0:55703:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514600/0x8e7c99ee62ab161e lrc: 1/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b8340c expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826424.564560:0:55703:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826424.564563:0:55703:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514600/0x8e7c99ee62ab161e lrc: 0/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b8340c expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000020:00100000:0.0:1787826424.564588:0:55703:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787826424.564601:0:55703:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aeed00 x1874670563980160/t0(0) o36->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826424.564629:0:55703:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55703:1874670563980160:192.168.203.139@tcp:36:lfs.0 00000100:00100000:0.0:1787826424.565105:0:55703:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e6c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826424.569033:0:55703:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55703:1874670563980160:192.168.203.139@tcp:36:lfs.0 00000080:00200000:1.0:1787826424.569171:0:55703:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 33 restore needed 0 00020000:00200000:1.0:1787826424.569201:0:55703:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1787826424.569222:0:55703:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:1.0:1787826424.569243:0:55703:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:1.0:1787826424.569249:0:55703:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e79230, osc lock ffff976883ff8ea0, flags 0x0 00010000:00010000:1.0:1787826424.569271:0:55703:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9768ad0ae800 type 11 mode 2 res 11811161089/269 (0 18446744073709551615) 00010000:00010000:1.0:1787826424.569329:0:55703:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000fdf26208/0x8e7c99ee62ab1664 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826424.569343:0:55703:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000fdf26208/0x8e7c99ee62ab1664 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826424.569354:0:55703:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000fdf26208/0x8e7c99ee62ab1664 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826424.569490:0:55703:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0161a40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55703:1874670563980288:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826424.569617:0:55703:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed36c0 going to sleep for 11 seconds 00000100:00100000:1.0:1787826424.572125:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826424.572234:0:53061:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:0.0:1787826424.572257:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585284992 00000100:00100000:0.0:1787826424.572268:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585284992 00000100:00100000:0.0:1787826424.572383:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 563 00000100:00100000:0.0:1787826424.572396:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768895b9450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8437:x1874670585284992:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826424.572419:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab164f lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b83459 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826424.572520:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826424.572527:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768895b9450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8437:x1874670585284992:12345-192.168.203.139@tcp:104: Request processed in 135us (415us total) trans 0 rc 0/0 00010000:00010000:3.0:1787826424.572724:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826424.572734:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab164f lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b83459 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826424.572744:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515e00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826424.572751:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab164f lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x22153cd396b83459 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1787826424.572775:0:42221:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787826424.572792:0:42221:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787826424.572890:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab164f lrc: 2/0,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b83459 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826424.572899:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826424.572929:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab164f lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b83459 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826424.572938:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab164f lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b83459 expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826424.572956:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.572958:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.572960:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826424.572968:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787826424.572987:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563980416:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826424.573233:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 563 00000100:00100000:1.0:1787826424.577488:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826424.577578:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585285120 00000100:00100000:0.0:1787826424.577589:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585285120 00000100:00100000:0.0:1787826424.577654:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 564 00000100:00100000:0.0:1787826424.577664:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0161e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:10341:x1874670585285120:12345-192.168.203.139@tcp:105: 00010000:00010000:0.0:1787826424.577684:0:42210:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000fdf26208/0x8e7c99ee62ab1664 lrc: 4/0,1 mode: --/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8348a expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.577699:0:42210:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000fdf26208/0x8e7c99ee62ab1664 lrc: 4/0,1 mode: --/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8348a expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.577717:0:42210:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000fdf26208/0x8e7c99ee62ab1664 lrc: 4/0,1 mode: --/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8348a expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.577730:0:42210:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000fdf26208/0x8e7c99ee62ab1664 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000000000 nid: local remote: 0x22153cd396b8348a expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.577742:0:42210:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000fdf26208/0x8e7c99ee62ab1664 lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000000000 nid: local remote: 0x22153cd396b8348a expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.577753:0:42210:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000fdf26208/0x8e7c99ee62ab1664 lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000000 nid: local remote: 0x22153cd396b8348a expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.577777:0:42210:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff976890fdf600) 00000100:00100000:0.0:1787826424.578100:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0161e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:10341:x1874670585285120:12345-192.168.203.139@tcp:105: Request processed in 436us (738us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826424.578110:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 564 00000100:00100000:3.0:1787826424.580401:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563980416:192.168.203.139@tcp:103:ldlm_bl.0 00000020:00010000:1.0:1787826424.585880:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885e73998 (ffff97688386c998/0) at cl_lock_release():245 00000020:00010000:1.0:1787826424.585885:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885e73998 (ffff97688386c998/0) at cl_lock_cancel():142 00000020:00010000:1.0:1787826424.585892:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e79230 (ffff976890d10908/1) at cl_lock_cancel():142 00010000:00010000:1.0:1787826424.585906:0:55703:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000fdf26208/0x8e7c99ee62ab1664 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8348a expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826424.585924:0:55703:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000fdf26208/0x8e7c99ee62ab1664 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8348a expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:1.0:1787826424.585944:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885e73998 (ffff97688386c998/0) at cl_lock_fini():72 00000020:00010000:1.0:1787826424.585947:0:55703:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e79230 (ffff97688386c998/1) at cl_lock_fini():72 00000080:00200000:1.0:1787826424.585969:0:55703:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 33 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1787826424.585985:0:55703:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8) current size 0, valid attrs 100101f0, mode 9768, uid 0, gid 0, new size 0, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826424.000000000, rc 0 00000080:00200080:1.0:1787826424.586022:0:55703:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=1be6570 00010000:00010000:1.0:1787826424.586084:0:55703:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfa00/0x8e7c99ee62ab163a lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0x22153cd396b83436 expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826424.586103:0:55703:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdfa00/0x8e7c99ee62ab163a lrc: 0/0,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x85809400000000 nid: local remote: 0x22153cd396b83436 expref: -99 pid: 55703 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787826424.586125:0:55703:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9768b0163480 x1874670563977728/t8589935194(8589935194) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826440 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1787826424.586143:0:55703:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1787826424.586149:0:55703:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b0160f00 x1874670563980800/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 504/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1787826424.586175:0:55703:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160f00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55703:1874670563980800:192.168.203.139@tcp:35:lfs.0 00000100:00100000:1.0:1787826424.586278:0:55703:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed36c0 going to sleep for 11 seconds 00000100:00100000:1.0:1787826424.588944:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826424.588970:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826424.589006:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585285248 00000100:00100000:1.0:1787826424.589016:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585285248 00000100:00100000:1.0:1787826424.589064:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585285376 00000100:00100000:1.0:1787826424.589066:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585285376 00000100:00100000:1.0:1787826424.589079:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 565 00000100:00100000:1.0:1787826424.589088:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0162d00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+6:14257:x1874670585285248:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826424.589110:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab15fb lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b833b1 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00010000:00010000:1.0:1787826424.589201:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826424.589208:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0162d00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+6:14257:x1874670585285248:12345-192.168.203.139@tcp:104: Request processed in 121us (277us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826424.589216:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 565 00000100:00100000:1.0:1787826424.589233:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 566 00000100:00100000:1.0:1787826424.589240:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0161680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585285376:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826424.589247:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab15e6 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83364 expref: -99 pid: 55695 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826424.589283:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:1.0:1787826424.589288:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0161680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585285376:12345-192.168.203.139@tcp:104: Request processed in 50us (318us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826424.589294:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 566 00010000:00010000:3.0:1787826424.589348:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1787826424.589357:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab15fb lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b833b1 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1787826424.589378:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515c00 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826424.589385:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab15fb lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b833b1 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1787826424.589394:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab15fb lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b833b1 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1787826424.589403:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab15fb lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 3 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b833b1 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1787826424.589435:0:42220:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab15e6 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83364 expref: -99 pid: 55695 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000002:00100000:3.0:1787826424.589491:0:42220:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889efa580 x1874670563975424/t8589935189(8589935189) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826440 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826424.589504:0:42220:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 2 is allocated opc 35, max 7 00000100:00100000:3.0:1787826424.589510:0:42220:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff97688cb5f0c0 x1874670563981056/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826424.589536:0:42220:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:42220:1874670563981056:192.168.203.139@tcp:35:ldlm_bl.0 00000100:00100000:3.0:1787826424.589594:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25540 going to sleep for 11 seconds 00000100:00100000:3.0:1787826424.590461:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563981184:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826424.598529:0:42220:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:42220:1874670563981056:192.168.203.139@tcp:35:ldlm_bl.0 00000002:00100000:3.0:1787826424.598595:0:42220:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889efa580 x1874670563975424/t8589935189(8589935189) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826440 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1787826424.598701:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab15fb lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b833b1 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1787826424.598710:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826424.598738:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab15fb lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b833b1 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1787826424.598747:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515c00/0x8e7c99ee62ab15fb lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b833b1 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1787826424.598776:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.598778:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.598780:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826424.598799:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563981312:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826424.608500:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563981312:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826424.609138:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826424.609767:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585285504 00000100:00100000:0.0:1787826424.609778:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585285504 00000100:00100000:0.0:1787826424.609876:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 567 00000100:00100000:0.0:1787826424.609889:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0162580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585285504:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826424.609910:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab1656 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x3e:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83475 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826424.610027:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826424.610034:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0162580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585285504:12345-192.168.203.139@tcp:104: Request processed in 148us (901us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826424.610044:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 567 00010000:00010000:3.0:1787826424.610227:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826424.610237:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab1656 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x3e:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83475 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826424.610247:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515a00 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826424.610252:0:42221:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab1656 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x3e:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83475 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826424.610261:0:42221:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab1656 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x3e:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83475 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826424.610275:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab1656 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x3e:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b83475 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826424.610304:0:42221:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x3e:0x0]: losing layout lock 00010000:00010000:3.0:1787826424.610427:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab1656 lrc: 2/0,0 mode: --/CR res: [0x200000405:0x3e:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83475 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826424.610436:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826424.611548:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab1656 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x3e:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83475 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826424.611563:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab1656 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x3e:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83475 expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826424.611592:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.611594:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.611597:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1787826424.710600:0:55709:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826424.710642:0:55709:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826424.710666:0:55709:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826424.710684:0:55709:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826424.710693:0:55709:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826424.710702:0:55709:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826424.710706:0:55709:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826424.710715:0:55709:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44225bbf8 00010000:00010000:0.0:1787826424.710720:0:55709:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826424.710730:0:55709:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826424.710749:0:55709:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826424.710759:0:55709:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=27 00000080:00200000:0.0:1787826424.710765:0:55709:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826424.710770:0:55709:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=267 00000080:00200000:0.0:1787826424.710801:0:55709:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826424.710809:0:55709:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826424.710816:0:55709:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826424.710872:0:55709:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826424.710879:0:55709:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826424.710883:0:55709:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826424.710885:0:55709:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826424.710889:0:55709:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44225bbf8 00010000:00010000:0.0:1787826424.710893:0:55709:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826424.710900:0:55709:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826424.710909:0:55709:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826424.710913:0:55709:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=267 00000080:00200000:0.0:1787826424.710917:0:55709:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 44 00000080:00200008:0.0:1787826424.710924:0:55709:0:(file.c:1090:ll_file_open()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00000080:00200080:0.0:1787826424.710961:0:55709:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8) cmd=c0086915 arg=21443d0 00000100:00100000:0.0:1787826424.711046:0:55709:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55709:1874670563982336:192.168.203.139@tcp:34:lfs.0 00000100:00100000:0.0:1787826424.711351:0:55709:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e540 going to sleep for 11 seconds 00000100:00100000:0.0:1787826424.714110:0:55709:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55709:1874670563982336:192.168.203.139@tcp:34:lfs.0 00000080:00200008:0.0:1787826424.714397:0:55709:0:(file.c:479:ll_file_release()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00010000:00010000:0.0:1787826424.714438:0:55709:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9600/0x8e7c99ee62ab0c1c lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b818c9 expref: -99 pid: 55483 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:3.0:1787826424.753971:0:55711:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826424.754026:0:55711:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826424.754051:0:55711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826424.754070:0:55711:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826424.754079:0:55711:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826424.754088:0:55711:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826424.754092:0:55711:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826424.754102:0:55711:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44223bbf8 00010000:00010000:3.0:1787826424.754210:0:55711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826424.754221:0:55711:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826424.754241:0:55711:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826424.754251:0:55711:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:3.0:1787826424.754258:0:55711:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787826424.754263:0:55711:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:3.0:1787826424.754267:0:55711:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 44 00000080:00200008:3.0:1787826424.754274:0:55711:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154400 00000002:00010000:3.0:1787826424.754294:0:55711:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114401 00010000:00010000:3.0:1787826424.754361:0:55711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1679 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 55711 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.754370:0:55711:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1679 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55711 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787826424.754384:0:55711:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787826424.754399:0:55711:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff97688cb5c000 x1874670563982592/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787826424.754409:0:55711:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1679 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55711 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826424.754437:0:55711:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55711:1874670563982592:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826424.754679:0:55711:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25900 going to sleep for 11 seconds 00000100:00100000:3.0:1787826424.758951:0:55711:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55711:1874670563982592:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826424.759001:0:55711:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1679 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b834e5 expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.759020:0:55711:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1679 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b834e5 expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.759034:0:55711:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1679 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b834e5 expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826424.759051:0:55711:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff97688cb5c000 x1874670563982592/t8589935203(8589935203) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826440 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826424.759064:0:55711:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff97688cb5c000 x1874670563982592/t8589935203(8589935203) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826440 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826424.759080:0:55711:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1679 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b834e5 expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826424.759095:0:55711:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1679 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b834e5 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.759238:0:55711:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826424.759248:0:55711:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826424.759308:0:55711:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x1d:0x0], updating i_size 0 00000080:00200000:3.0:1787826424.759323:0:55711:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 34 -> 34 00000080:00010000:3.0:1787826424.759328:0:55711:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1679 00000080:00010000:3.0:1787826424.759334:0:55711:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1679 from it ffffb2d44223bbf0 00010000:00010000:3.0:1787826424.759339:0:55711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1679 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b834e5 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.759350:0:55711:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1679 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b834e5 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000002:00100000:3.0:1787826424.759385:0:55711:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff97688cb5c000 x1874670563982592/t8589935203(8589935203) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826440 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1787826424.759575:0:55711:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:3.0:1787826424.759585:0:55711:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:3.0:1787826424.759617:0:55711:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826424.759625:0:55711:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:3.0:1787826424.759654:0:55711:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826424.759667:0:55711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515600/0x8e7c99ee62ab1672 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b834c2 expref: -99 pid: 55705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826424.759680:0:55711:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515600/0x8e7c99ee62ab1672 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b834c2 expref: -99 pid: 55705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826424.759689:0:55711:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826424.759694:0:55711:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:3.0:1787826424.759698:0:55711:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1672 00000080:00010000:3.0:1787826424.759703:0:55711:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1672 from it ffffb2d44223bcd8 00010000:00010000:3.0:1787826424.759707:0:55711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515600/0x8e7c99ee62ab1672 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b834c2 expref: -99 pid: 55705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826424.759715:0:55711:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515600/0x8e7c99ee62ab1672 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b834c2 expref: -99 pid: 55705 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:3.0:1787826424.759772:0:55711:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154000 00010000:00010000:3.0:1787826424.759786:0:55711:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1679 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b834e5 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826424.760179:0:55711:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826424.760202:0:55711:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826424.760212:0:55711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826424.760222:0:55711:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826424.760229:0:55711:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826424.760234:0:55711:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826424.760237:0:55711:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826424.760242:0:55711:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44223bbf8 00010000:00010000:3.0:1787826424.760245:0:55711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826424.760253:0:55711:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826424.760264:0:55711:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826424.760272:0:55711:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:3.0:1787826424.760278:0:55711:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787826424.760283:0:55711:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:3.0:1787826424.760288:0:55711:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 44 00000080:00200008:3.0:1787826424.760296:0:55711:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154400 00000080:00200000:3.0:1787826424.760337:0:55711:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:3.0:1787826424.760342:0:55711:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:3.0:1787826424.760361:0:55711:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826424.760365:0:55711:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:3.0:1787826424.760375:0:55711:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826424.760383:0:55711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515600/0x8e7c99ee62ab1672 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b834c2 expref: -99 pid: 55705 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826424.760391:0:55711:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515600/0x8e7c99ee62ab1672 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b834c2 expref: -99 pid: 55705 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826424.760396:0:55711:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826424.760400:0:55711:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:3.0:1787826424.760403:0:55711:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1672 00000080:00010000:3.0:1787826424.760407:0:55711:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1672 from it ffffb2d44223bcd8 00010000:00010000:3.0:1787826424.760410:0:55711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515600/0x8e7c99ee62ab1672 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b834c2 expref: -99 pid: 55705 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826424.760417:0:55711:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515600/0x8e7c99ee62ab1672 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b834c2 expref: -99 pid: 55705 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200008:3.0:1787826424.760682:0:55711:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154000 00010000:00010000:3.0:1787826424.760695:0:55711:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1679 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b834e5 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826424.760747:0:55711:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826424.760756:0:55711:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826424.760763:0:55711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826424.760771:0:55711:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826424.760777:0:55711:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826424.760781:0:55711:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826424.760784:0:55711:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826424.760787:0:55711:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44223bbf8 00010000:00010000:3.0:1787826424.760790:0:55711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826424.760796:0:55711:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826424.760804:0:55711:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826424.760808:0:55711:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:3.0:1787826424.760811:0:55711:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787826424.760814:0:55711:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:3.0:1787826424.760816:0:55711:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 46 00000080:00200008:3.0:1787826424.760820:0:55711:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154402 00000002:00010000:3.0:1787826424.763254:0:55711:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114403 00010000:00010000:3.0:1787826424.763345:0:55711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1680 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 55711 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.763357:0:55711:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1680 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55711 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787826424.763376:0:55711:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787826424.763393:0:55711:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff97688cb5f0c0 x1874670563982848/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787826424.763403:0:55711:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1680 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55711 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826424.763431:0:55711:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55711:1874670563982848:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826424.765252:0:55711:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25900 going to sleep for 11 seconds 00000100:00100000:0.0:1787826424.778599:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826424.778723:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585286016 00000100:00100000:1.0:1787826424.778735:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585286016 00000100:00100000:1.0:1787826424.778886:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 568 00000100:00100000:1.0:1787826424.778898:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aed680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6518:x1874670585286016:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826424.778921:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515600/0x8e7c99ee62ab1672 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b834c2 expref: -99 pid: 55705 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787826424.779021:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826424.779028:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aed680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6518:x1874670585286016:12345-192.168.203.139@tcp:104: Request processed in 132us (444us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826424.779036:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 568 00000100:00100000:3.0:1787826424.783283:0:55711:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55711:1874670563982848:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826424.783323:0:55711:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode CW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1680 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b834fa expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.783339:0:55711:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1680 lrc: 4/1,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b834fa expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.783355:0:55711:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1680 lrc: 4/1,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b834fa expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.783685:0:55711:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1680 lrc: 4/1,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b834fa expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.783704:0:55711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1680 lrc: 5/1,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b834fa expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.783713:0:55711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1680 lrc: 5/1,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b834fa expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.783728:0:55711:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1680 lrc: 4/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b834fa expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826424.783740:0:55711:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff97688cb5f0c0 x1874670563982848/t8589935204(8589935204) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826440 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826424.783754:0:55711:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff97688cb5f0c0 x1874670563982848/t8589935204(8589935204) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826440 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826424.783772:0:55711:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1680 lrc: 3/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b834fa expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.783788:0:55711:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 8 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826424.783794:0:55711:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826424.783878:0:55711:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x1d:0x0], updating i_size 0 00000080:00010000:3.0:1787826424.783884:0:55711:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1680 00000080:00010000:3.0:1787826424.783893:0:55711:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1680 from it ffffb2d44223bbf0 00010000:00010000:3.0:1787826424.783896:0:55711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1680 lrc: 3/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b834fa expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.783905:0:55711:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1680 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b834fa expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:3.0:1787826424.783942:0:55711:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff97688cb5f0c0 x1874670563982848/t8589935204(8589935204) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826440 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:3.0:1787826424.784131:0:55711:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=800466af arg=7fff0bed3fac 00000080:00200080:3.0:1787826424.784150:0:55711:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=80086601 arg=7fff0bed3f48 00000100:00100000:3.0:1787826424.784211:0:55711:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55711:1874670563983104:192.168.203.139@tcp:33:lfs.0 00000100:00100000:3.0:1787826424.784309:0:55711:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25900 going to sleep for 11 seconds 00000100:00100000:3.0:1787826424.786584:0:55711:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55711:1874670563983104:192.168.203.139@tcp:33:lfs.0 00000080:00200080:3.0:1787826424.786632:0:55711:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=7fff0bed3f34 00000002:00010000:3.0:1787826424.786670:0:55711:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 01004040000002 00010000:00010000:3.0:1787826424.786689:0:55711:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1680 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x22153cd396b834fa expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.786720:0:55711:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1679 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b834e5 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826424.786731:0:55711:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab166b lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8349f expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000002:00100000:3.0:1787826424.786770:0:55711:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff97688cb5c000 x1874670563982592/t8589935203(8589935203) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826440 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826424.786781:0:55711:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1787826424.786787:0:55711:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff97688cb5d680 x1874670563983360/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826424.786812:0:55711:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55711:1874670563983360:192.168.203.139@tcp:35:lfs.0 00000100:00100000:3.0:1787826424.787058:0:55711:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25900 going to sleep for 11 seconds 00000100:00100000:3.0:1787826424.791741:0:55711:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55711:1874670563983360:192.168.203.139@tcp:35:lfs.0 00000002:00100000:3.0:1787826424.791785:0:55711:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff97688cb5c000 x1874670563982592/t8589935203(8589935203) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826440 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1787826424.792412:0:55711:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1680 lrc: 1/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b834fa expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.792427:0:55711:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1679 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b834e5 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826424.792434:0:55711:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1787826424.792436:0:55711:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8a00/0x8e7c99ee62ab1680 lrc: 0/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b834fa expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826424.792454:0:55711:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1679 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b834e5 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826424.792483:0:55711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1687 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 55711 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.792489:0:55711:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x81000000001000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1687 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 55711 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787826424.792501:0:55711:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787826424.792508:0:55711:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff97688cb5d680 x1874670563983616/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 976/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787826424.792518:0:55711:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1687 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x81000000000000 pid: 55711 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826424.792543:0:55711:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55711:1874670563983616:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826424.792780:0:55711:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25900 going to sleep for 11 seconds 00000100:00100000:3.0:1787826424.797516:0:55711:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55711:1874670563983616:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826424.797558:0:55711:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1687 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b8350f expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.797579:0:55711:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1687 lrc: 4/1,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b8350f expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.797595:0:55711:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1687 lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b8350f expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.797606:0:55711:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1687 lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b8350f expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.797621:0:55711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1687 lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b8350f expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.797630:0:55711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1687 lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b8350f expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.797648:0:55711:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1687 lrc: 4/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b8350f expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826424.797661:0:55711:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x600002b, status=0 req@ffff97688cb5d680 x1874670563983616/t8589935206(8589935206) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 976/848 e 0 to 0 dl 1787826440 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826424.797674:0:55711:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff97688cb5d680 x1874670563983616/t8589935206(8589935206) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 976/848 e 0 to 0 dl 1787826440 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826424.797691:0:55711:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1687 lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b8350f expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.797712:0:55711:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 2 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826424.797720:0:55711:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000002:00100000:3.0:1787826424.797744:0:55711:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff97688cb5d680 x1874670563983616/t8589935206(8589935206) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 768/848 e 0 to 0 dl 1787826440 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1787826424.797753:0:55711:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1687 00000080:00010000:3.0:1787826424.797769:0:55711:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1687 from it ffffb2d44223bcc0 00010000:00010000:3.0:1787826424.797773:0:55711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1687 lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b8350f expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.797780:0:55711:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8800/0x8e7c99ee62ab1687 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b8350f expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826424.805196:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826424.805322:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585286144 00000100:00100000:0.0:1787826424.805335:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585286144 00000100:00100000:0.0:1787826424.805447:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 569 00000100:00100000:0.0:1787826424.805461:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0163c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6522:x1874670585286144:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826424.805486:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab166b lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b8349f expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826424.805752:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826424.805767:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0163c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6522:x1874670585286144:12345-192.168.203.139@tcp:104: Request processed in 303us (593us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826424.805777:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 569 00010000:00010000:3.0:1787826424.806125:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826424.806136:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab166b lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b8349f expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826424.806147:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515200 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826424.806154:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab166b lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b8349f expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826424.806164:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab166b lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b8349f expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826424.806174:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab166b lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b8349f expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1787826424.806221:0:42220:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x1d:0x0]: losing layout lock 00010000:00010000:3.0:1787826424.806273:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab166b lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b8349f expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826424.806281:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826424.806496:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab166b lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b8349f expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826424.806505:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab166b lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b8349f expref: -99 pid: 55703 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826424.806531:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.806533:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.806535:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826424.841559:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826424.841654:0:53061:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:0.0:1787826424.841679:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585286272 00000100:00100000:0.0:1787826424.841690:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585286272 00000100:00100000:0.0:1787826424.841759:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 570 00000100:00100000:0.0:1787826424.841772:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768895ba850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585286272:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826424.841796:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515600/0x8e7c99ee62ab168e lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83524 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826424.841967:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826424.841975:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768895ba850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585286272:12345-192.168.203.139@tcp:104: Request processed in 206us (429us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826424.841984:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 570 00010000:00010000:3.0:1787826424.842053:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826424.842062:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515600/0x8e7c99ee62ab168e lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83524 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826424.842072:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515600 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826424.842078:0:42221:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515600/0x8e7c99ee62ab168e lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83524 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826424.842087:0:42221:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515600/0x8e7c99ee62ab168e lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83524 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826424.842095:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515600/0x8e7c99ee62ab168e lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b83524 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826424.842131:0:42221:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x1d:0x0]: losing layout lock 00010000:00010000:3.0:1787826424.842183:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515600/0x8e7c99ee62ab168e lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83524 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826424.842203:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826424.842241:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515600/0x8e7c99ee62ab168e lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83524 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826424.842250:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515600/0x8e7c99ee62ab168e lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83524 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826424.842277:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.842279:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.842281:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826424.842291:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826424.842367:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563985024:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826424.844265:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563985024:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826424.845549:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826424.845681:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585286400 00000100:00100000:0.0:1787826424.845691:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585286400 00000100:00100000:0.0:1787826424.845758:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 571 00000100:00100000:0.0:1787826424.845769:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b01603c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585286400:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826424.845792:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab1695 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83532 expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826424.846132:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826424.846140:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b01603c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585286400:12345-192.168.203.139@tcp:104: Request processed in 373us (1131us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826424.846149:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 571 00000080:00200000:2.0:1787826424.846808:0:49796:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x3f:0x0]: losing layout lock 00010000:00010000:2.0:1787826424.846970:0:49796:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab1695 lrc: 2/0,0 mode: --/PR res: [0x200000405:0x3f:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83532 expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826424.846978:0:49796:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1787826424.847035:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563985152:192.168.203.139@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1787826424.847651:0:49796:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab1695 lrc: 1/0,0 mode: --/PR res: [0x200000405:0x3f:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83532 expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826424.847664:0:49796:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab1695 lrc: 0/0,0 mode: --/PR res: [0x200000405:0x3f:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83532 expref: -99 pid: 55711 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826424.847692:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826424.847694:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826424.847696:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826424.854301:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563985152:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826424.863225:0:55711:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55711:1874670563984896:192.168.203.139@tcp:35:lfs.0 00000002:00100000:2.0:1787826424.863284:0:55711:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889ef8f00 x1874670563984640/t8589935209(8589935209) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826440 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1787826424.863342:0:55711:0:(file.c:7076:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000002:00010000:2.0:1787826424.863357:0:55711:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: layout flags 0 00010000:00010000:2.0:1787826424.863423:0:55711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55711 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826424.863435:0:55711:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55711 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826424.863446:0:55711:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55711 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826424.863473:0:55711:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55711:1874670563985280:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826424.863568:0:55711:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35840 going to sleep for 11 seconds 00000100:00100000:2.0:1787826424.867296:0:55711:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55711:1874670563985280:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826424.867341:0:55711:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83563 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826424.867426:0:55711:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83563 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826424.867448:0:55711:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff976889efb0c0 x1874670563985280/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 376/480 e 0 to 0 dl 1787826440 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826424.867470:0:55711:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83563 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787826424.867985:0:55711:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83563 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826424.868002:0:55711:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:2.0:1787826424.868010:0:55711:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1787826424.868034:0:55711:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab16a3 00000080:00010000:2.0:1787826424.868044:0:55711:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab16a3 from it ffffb2d44223bbc8 00010000:00010000:2.0:1787826424.868048:0:55711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83563 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826424.868059:0:55711:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83563 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826424.868091:0:55711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83563 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826424.868110:0:55711:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83563 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1787826424.868119:0:55711:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83563 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826424.868165:0:55711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83563 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826424.868173:0:55711:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83563 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826424.868204:0:55711:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787826424.868217:0:55711:0:(lov_io.c:693:lov_io_mirror_init()) [0x200000405:0x1d:0x0]: flr state: 1, move mirror from 1 to 1, have retried: 0, mirror count: 2 00020000:00200000:2.0:1787826424.868221:0:55711:0:(lov_io.c:750:lov_io_mirror_init()) use delayed RPC state for this IO 00020000:00200000:2.0:1787826424.868229:0:55711:0:(lov_io.c:1253:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:2.0:1787826424.868265:0:55711:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:2.0:1787826424.868279:0:55711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83563 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826424.868287:0:55711:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83563 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1787826424.868294:0:55711:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83563 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826424.868311:0:55711:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787826424.868316:0:55711:0:(lov_io.c:693:lov_io_mirror_init()) [0x200000405:0x1d:0x0]: flr state: 1, move mirror from 1 to 1, have retried: 0, mirror count: 2 00020000:00200000:2.0:1787826424.868319:0:55711:0:(lov_io.c:750:lov_io_mirror_init()) use delayed RPC state for this IO 00020000:00200000:2.0:1787826424.868321:0:55711:0:(lov_io.c:1253:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:2.0:1787826424.868330:0:55711:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787826424.868426:0:55711:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 4294967294 -> 35 00010000:00010000:2.0:1787826424.868431:0:55711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83563 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826424.868439:0:55711:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83563 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:2.0:1787826424.869023:0:55711:0:(file.c:479:ll_file_release()) START file . :VOLATILE:0000:7D36B029:fd=03:[0x200000405:0x3f:0x0](ffff976890b3b248), flags 554002 00000002:00100000:2.0:1787826424.869092:0:55711:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889ef9680 x1874670563984384/t8589935208(8589935208) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826440 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787826424.869105:0:55711:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787826424.869112:0:55711:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efb0c0 x1874670563985536/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787826424.869139:0:55711:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55711:1874670563985536:192.168.203.139@tcp:35:lfs.0 00000100:00100000:2.0:1787826424.869261:0:55711:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35180 going to sleep for 11 seconds 00000100:00100000:2.0:1787826424.872000:0:55711:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55711:1874670563985536:192.168.203.139@tcp:35:lfs.0 00000002:00100000:2.0:1787826424.872123:0:55711:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889ef9680 x1874670563984384/t8589935208(8589935208) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826440 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1787826424.872210:0:55711:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154002 00000002:00100000:2.0:1787826424.872280:0:55711:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff97688cb5f0c0 x1874670563982848/t8589935204(8589935204) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826440 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787826424.872296:0:55711:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787826424.872302:0:55711:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889ef8f00 x1874670563985792/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787826424.872326:0:55711:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8f00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55711:1874670563985792:192.168.203.139@tcp:35:lfs.0 00000100:00100000:2.0:1787826424.872488:0:55711:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35180 going to sleep for 11 seconds 00000100:00100000:2.0:1787826424.877247:0:55711:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8f00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55711:1874670563985792:192.168.203.139@tcp:35:lfs.0 00000002:00100000:2.0:1787826424.877298:0:55711:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff97688cb5f0c0 x1874670563982848/t8589935204(8589935204) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826440 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1787826424.906887:0:55713:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826424.906927:0:55713:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826424.906952:0:55713:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826424.906971:0:55713:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826424.906985:0:55713:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826424.906996:0:55713:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826424.906999:0:55713:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826424.907009:0:55713:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d442213b58 00010000:00010000:1.0:1787826424.907025:0:55713:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826424.907034:0:55713:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826424.907055:0:55713:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826424.907065:0:55713:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=24 00000080:00200000:1.0:1787826424.907073:0:55713:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826424.907088:0:55713:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), flags=8 00000080:00200000:1.0:1787826424.907093:0:55713:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=getattr 00000002:00010000:1.0:1787826424.907110:0:55713:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f60b.sanity-flr,[0x0:0x0:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787826424.907178:0:55713:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55713 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826424.907199:0:55713:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55713 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826424.907213:0:55713:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55713 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826424.907249:0:55713:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0161e00 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55713:1874670563985920:192.168.203.139@tcp:101:getfattr.0 00000100:00100000:1.0:1787826424.907545:0:55713:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3180 going to sleep for 11 seconds 00000100:00100000:2.0:1787826424.910315:0:55713:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0161e00 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55713:1874670563985920:192.168.203.139@tcp:101:getfattr.0 00010000:00010000:2.0:1787826424.910347:0:55713:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 4/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826424.910377:0:55713:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826424.910387:0:55713:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826424.910401:0:55713:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826424.910415:0:55713:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826424.910431:0:55713:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826424.910441:0:55713:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826424.910449:0:55713:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826424.910462:0:55713:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9768b0161e00 x1874670563985920/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 584/720 e 0 to 0 dl 1787826440 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'getfattr.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826424.910476:0:55713:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787826424.910492:0:55713:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826424.910504:0:55713:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/29 (0 0) 00000080:00200000:2.0:1787826424.910513:0:55713:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787826424.910569:0:55713:0:(namei.c:147:ll_iget()) got inode: [0x200000405:0x1d:0x0](ffff9768901a80c8): rc = 0 00000080:00200000:2.0:1787826424.910582:0:55713:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 35 -> 35 00000080:00010000:2.0:1787826424.910588:0:55713:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab16aa 00000080:00010000:2.0:1787826424.910601:0:55713:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:2.0:1787826424.910605:0:55713:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab16aa from it ffffb2d442213b20 00010000:00010000:2.0:1787826424.910608:0:55713:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826424.910618:0:55713:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1787826424.910668:0:55713:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787826424.910676:0:55713:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:2.0:1787826424.910705:0:55713:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826424.910716:0:55713:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826424.910728:0:55713:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826424.910736:0:55713:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826424.910743:0:55713:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:2.0:1787826424.910746:0:55713:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab16aa 00000080:00010000:2.0:1787826424.910750:0:55713:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab16aa from it ffffb2d442213cb8 00010000:00010000:2.0:1787826424.910753:0:55713:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826424.910761:0:55713:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826424.910782:0:55713:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787826424.910796:0:55713:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:2.0:1787826424.910818:0:55713:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787826424.910859:0:55713:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff97688757cdb0, osc lock ffff976883f7a9c0, flags 0x400001002 00010000:00010000:2.0:1787826424.910881:0:55713:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826424.910899:0:55713:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826424.910910:0:55713:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826424.910919:0:55713:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826424.910961:0:55713:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab16b1 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55713 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826424.910969:0:55713:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab16b1 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55713 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1787826424.911008:0:55713:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1787826424.911408:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563986048:192.168.203.139@tcp:101:getfattr.0 00000100:00100000:1.0:1787826424.915312:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826424.915905:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585286656 00000100:00100000:0.0:1787826424.915919:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585286656 00000100:00100000:0.0:1787826424.915992:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 572 00000100:00100000:0.0:1787826424.916003:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0162d00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8435:x1874670585286656:12345-192.168.203.139@tcp:106: 00010000:00010000:0.0:1787826424.916021:0:53061:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826424.916036:0:53061:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:0.0:1787826424.916068:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@000000000e631c06[0x110, 1, [0x200000405:0x3e:0x0] hash] 00000020:00010000:0.0:1787826424.916070:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826424 blocks: 0 00000020:00010000:0.0:1787826424.916074:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@000000000964cd23[0x116, 2, [0x280000401:0x10d:0x0]] 00000020:00010000:0.0:1787826424.916076:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826424 blocks: 0 00000100:00100000:0.0:1787826424.916160:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0162d00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8435:x1874670585286656:12345-192.168.203.139@tcp:106: Request processed in 157us (862us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826424.916168:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 572 00010000:00010000:3.0:1787826424.919578:0:2404:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab16b1 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55713 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.919605:0:2404:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab16b1 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55713 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.919622:0:2404:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab16b1 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55713 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.919630:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab16b1 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55713 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.919651:0:2404:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab16b1 lrc: 5/1,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55713 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826424.919668:0:2404:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:3.0:1787826424.919676:0:2404:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826424.919694:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab16b1 lrc: 4/1,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55713 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.919703:0:2404:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab16b1 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55713 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.919711:0:2404:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab16b1 lrc: 4/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55713 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.919718:0:2404:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515a00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826424.919725:0:2404:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab16b1 lrc: 5/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55713 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.919742:0:2404:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826424.919751:0:2404:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab16b1 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55713 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.919765:0:2404:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004f154db8/0x8e7c99ee62ab16b1 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55713 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826424.919790:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563986048:192.168.203.139@tcp:101:getfattr.0 00000080:00200000:2.0:1787826424.920282:0:55713:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1787826424.920292:0:55713:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf3b40 (ffff976890d10ea8/0) at cl_lock_release():245 00000020:00010000:2.0:1787826424.920297:0:55713:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf3b40 (ffff976890d10ea8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787826424.920303:0:55713:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757cdb0 (ffff976890d106c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787826424.920308:0:55713:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf3b40 (ffff976890d10ea8/0) at cl_lock_fini():72 00000020:00010000:2.0:1787826424.920316:0:55713:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757cdb0 (ffff976890d10ea8/1) at cl_lock_fini():72 00000080:00200000:2.0:1787826424.920334:0:55713:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 35 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787826424.920396:0:55713:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:2.0:1787826424.920426:0:55713:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826424.920449:0:55713:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826424.920464:0:55713:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826424.920474:0:55713:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826424.920482:0:55713:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:2.0:1787826424.920486:0:55713:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:2.0:1787826424.920494:0:55713:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d442213bf8 00010000:00010000:2.0:1787826424.920498:0:55713:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826424.920507:0:55713:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826424.920525:0:55713:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:2.0:1787826424.920533:0:55713:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:2.0:1787826424.920540:0:55713:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787826424.920544:0:55713:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=9 00000080:00200000:2.0:1787826424.920553:0:55713:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:2.0:1787826424.920560:0:55713:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200000:2.0:1787826424.920583:0:55713:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:2.0:1787826424.920591:0:55713:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826424.920598:0:55713:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826424.920607:0:55713:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826424.920614:0:55713:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826424.920618:0:55713:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:2.0:1787826424.920621:0:55713:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:2.0:1787826424.920625:0:55713:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d442213bf8 00010000:00010000:2.0:1787826424.920629:0:55713:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826424.920637:0:55713:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826424.920647:0:55713:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:2.0:1787826424.920651:0:55713:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:2.0:1787826424.920654:0:55713:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787826424.920657:0:55713:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=9 00000080:00200000:2.0:1787826424.920666:0:55713:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:2.0:1787826424.920670:0:55713:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200000:1.0:1787826424.941505:0:55717:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826424.941749:0:55717:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826424.941776:0:55717:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826424.941795:0:55717:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826424.941806:0:55717:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826424.941817:0:55717:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826424.941820:0:55717:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826424.941916:0:55717:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44223bbf8 00010000:00010000:1.0:1787826424.941921:0:55717:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826424.941931:0:55717:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826424.941950:0:55717:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826424.941959:0:55717:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=27 00000080:00200000:1.0:1787826424.941966:0:55717:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826424.941971:0:55717:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=267 00000080:00200000:1.0:1787826424.942009:0:55717:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826424.942017:0:55717:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826424.942024:0:55717:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826424.942032:0:55717:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826424.942040:0:55717:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826424.942044:0:55717:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826424.942048:0:55717:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826424.942052:0:55717:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44223bbf8 00010000:00010000:1.0:1787826424.942055:0:55717:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826424.942063:0:55717:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826424.942071:0:55717:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826424.942074:0:55717:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=267 00000080:00200000:1.0:1787826424.942078:0:55717:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 44 00000080:00200008:1.0:1787826424.942084:0:55717:0:(file.c:1090:ll_file_open()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00000080:00200080:1.0:1787826424.942118:0:55717:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8) cmd=c0086915 arg=bb53d0 00000100:00100000:1.0:1787826424.942215:0:55717:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0161680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55717:1874670563986176:192.168.203.139@tcp:34:lfs.0 00000100:00100000:1.0:1787826424.942391:0:55717:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed39c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826424.950127:0:55717:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0161680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55717:1874670563986176:192.168.203.139@tcp:34:lfs.0 00000080:00200008:3.0:1787826424.950506:0:55717:0:(file.c:479:ll_file_release()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00010000:00010000:3.0:1787826424.950658:0:55717:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9600/0x8e7c99ee62ab0c1c lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b818c9 expref: -99 pid: 55483 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:0.0:1787826424.977489:0:55719:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826424.977530:0:55719:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826424.977552:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826424.977645:0:55719:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826424.977655:0:55719:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826424.977666:0:55719:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826424.977671:0:55719:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826424.977680:0:55719:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44225bbf8 00010000:00010000:0.0:1787826424.977684:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826424.977694:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826424.977714:0:55719:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826424.977725:0:55719:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:0.0:1787826424.977732:0:55719:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826424.977736:0:55719:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:0.0:1787826424.977740:0:55719:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 44 00000080:00200008:0.0:1787826424.977745:0:55719:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154400 00000002:00010000:0.0:1787826424.977764:0:55719:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114401 00010000:00010000:0.0:1787826424.977959:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab16b8 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826424.977968:0:55719:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab16b8 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826424.977984:0:55719:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826424.977999:0:55719:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aed680 x1874670563986432/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826424.978009:0:55719:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab16b8 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826424.978035:0:55719:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563986432:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826424.978326:0:55719:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e240 going to sleep for 11 seconds 00000100:00100000:2.0:1787826424.983660:0:55719:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563986432:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826424.983699:0:55719:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab16b8 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b835b0 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826424.983718:0:55719:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab16b8 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b835b0 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826424.983731:0:55719:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab16b8 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b835b0 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826424.983754:0:55719:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff976882aed680 x1874670563986432/t8589935216(8589935216) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826440 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787826424.983770:0:55719:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aed680 x1874670563986432/t8589935216(8589935216) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826440 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826424.983785:0:55719:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab16b8 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b835b0 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787826424.983803:0:55719:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab16b8 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b835b0 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826424.983818:0:55719:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:2.0:1787826424.983854:0:55719:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787826424.983924:0:55719:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 35 -> 35 00000080:00010000:2.0:1787826424.983930:0:55719:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab16b8 00000080:00010000:2.0:1787826424.983938:0:55719:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab16b8 from it ffffb2d44225bbf0 00010000:00010000:2.0:1787826424.983943:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab16b8 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b835b0 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826424.983954:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab16b8 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b835b0 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000002:00100000:2.0:1787826424.983988:0:55719:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aed680 x1874670563986432/t8589935216(8589935216) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826440 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1787826424.984172:0:55719:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:2.0:1787826424.984182:0:55719:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:2.0:1787826424.984226:0:55719:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787826424.984235:0:55719:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:2.0:1787826424.984262:0:55719:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826424.984276:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826424.984288:0:55719:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826424.984296:0:55719:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826424.984302:0:55719:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:2.0:1787826424.984306:0:55719:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab16aa 00000080:00010000:2.0:1787826424.984310:0:55719:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab16aa from it ffffb2d44225bcd8 00010000:00010000:2.0:1787826424.984313:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826424.984320:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787826424.984340:0:55719:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 35 restore needed 0 00000001:00010000:2.0:1787826424.984353:0:55719:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:2.0:1787826424.984390:0:55719:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787826424.984395:0:55719:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff97688757cdb0, osc lock ffff976883f7a9c0, flags 0x400001002 00010000:00010000:2.0:1787826424.984415:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826424.984426:0:55719:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826424.984441:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826424.984448:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826424.984494:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000053d2415/0x8e7c99ee62ab16bf lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826424.984503:0:55719:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000053d2415/0x8e7c99ee62ab16bf lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1787826424.984535:0:55719:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1787826424.985055:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563986560:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826424.987596:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826424.987650:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585286784 00000100:00100000:0.0:1787826424.987662:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585286784 00000100:00100000:0.0:1787826424.987733:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 573 00000100:00100000:0.0:1787826424.987745:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aecf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:14864:x1874670585286784:12345-192.168.203.139@tcp:106: 00010000:00010000:0.0:1787826424.987766:0:42211:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826424.987782:0:42211:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00010000:0.0:1787826424.987816:0:42211:0:(cl_object.c:322:cl_object_glimpse()) header@000000000e631c06[0x110, 1, [0x200000405:0x3e:0x0] hash] 00000020:00010000:0.0:1787826424.987818:0:42211:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826424 blocks: 0 00000020:00010000:0.0:1787826424.987854:0:42211:0:(cl_object.c:322:cl_object_glimpse()) header@000000000964cd23[0x116, 2, [0x280000401:0x10d:0x0]] 00000020:00010000:0.0:1787826424.987856:0:42211:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826424 blocks: 0 00000100:00100000:0.0:1787826424.987924:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aecf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:14864:x1874670585286784:12345-192.168.203.139@tcp:106: Request processed in 181us (342us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826424.987933:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 573 00010000:00010000:3.0:1787826424.991357:0:2403:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000053d2415/0x8e7c99ee62ab16bf lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.991395:0:2403:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000053d2415/0x8e7c99ee62ab16bf lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.991412:0:2403:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000053d2415/0x8e7c99ee62ab16bf lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.991428:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000053d2415/0x8e7c99ee62ab16bf lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.991437:0:2403:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000053d2415/0x8e7c99ee62ab16bf lrc: 5/1,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826424.991452:0:2403:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:3.0:1787826424.991460:0:2403:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826424.991476:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000053d2415/0x8e7c99ee62ab16bf lrc: 4/1,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.991485:0:2403:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000053d2415/0x8e7c99ee62ab16bf lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.991494:0:2403:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000053d2415/0x8e7c99ee62ab16bf lrc: 4/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.991501:0:2403:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515c00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826424.991508:0:2403:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000053d2415/0x8e7c99ee62ab16bf lrc: 5/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.991523:0:2403:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826424.991531:0:2403:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000053d2415/0x8e7c99ee62ab16bf lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826424.991554:0:2403:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000053d2415/0x8e7c99ee62ab16bf lrc: 0/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826424.991584:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563986560:192.168.203.139@tcp:101:lfs.0 00000080:00200000:2.0:1787826424.991738:0:55719:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1787826424.991746:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf3b40 (ffff976890d10ea8/0) at cl_lock_release():245 00000020:00010000:2.0:1787826424.991750:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf3b40 (ffff976890d10ea8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787826424.991756:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757cdb0 (ffff976890d106c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787826424.991761:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf3b40 (ffff976890d10ea8/0) at cl_lock_fini():72 00000020:00010000:2.0:1787826424.991764:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757cdb0 (ffff976890d10ea8/1) at cl_lock_fini():72 00000080:00200000:2.0:1787826424.991781:0:55719:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 35 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1787826424.991902:0:55719:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154000 00010000:00010000:2.0:1787826424.991940:0:55719:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab16b8 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b835b0 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826424.992071:0:55719:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:2.0:1787826424.992096:0:55719:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826424.992109:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826424.992119:0:55719:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826424.992127:0:55719:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826424.992137:0:55719:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:2.0:1787826424.992141:0:55719:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:2.0:1787826424.992148:0:55719:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44225bbf8 00010000:00010000:2.0:1787826424.992151:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826424.992159:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826424.992174:0:55719:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:2.0:1787826424.992182:0:55719:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:2.0:1787826424.992201:0:55719:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787826424.992206:0:55719:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:2.0:1787826424.992210:0:55719:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 46 00000080:00200008:2.0:1787826424.992220:0:55719:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 1154402 00000002:00010000:2.0:1787826424.992238:0:55719:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114403 00010000:00010000:2.0:1787826424.992295:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515e00/0x8e7c99ee62ab16c6 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826424.992302:0:55719:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515e00/0x8e7c99ee62ab16c6 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1787826424.992319:0:55719:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1787826424.992331:0:55719:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efbc00 x1874670563986816/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1787826424.992341:0:55719:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515e00/0x8e7c99ee62ab16c6 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826424.992368:0:55719:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563986816:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826424.992726:0:55719:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35240 going to sleep for 11 seconds 00000100:00100000:0.0:1787826424.996908:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826424.997027:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585286912 00000100:00100000:1.0:1787826424.997039:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585286912 00000100:00100000:1.0:1787826424.997110:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 574 00000100:00100000:1.0:1787826424.997120:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aec3c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6519:x1874670585286912:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826424.997143:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826424.997247:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826424.997255:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aec3c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6519:x1874670585286912:12345-192.168.203.139@tcp:104: Request processed in 137us (362us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826424.997263:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 574 00010000:00010000:3.0:1787826424.997382:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826424.997391:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826424.997400:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976890fde800 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826424.997407:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826424.997416:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826424.997426:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826424.997459:0:42220:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab16b8 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b835b0 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826424.997512:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 2/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826424.997520:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826424.997565:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 1/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826424.997574:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde800/0x8e7c99ee62ab16aa lrc: 0/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83586 expref: -99 pid: 55713 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826424.997592:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.997594:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826424.997596:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826424.997606:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826424.997624:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563986944:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826425.001251:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563986944:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826425.003302:0:55719:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563986816:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826425.003337:0:55719:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode CW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515e00/0x8e7c99ee62ab16c6 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b835cc expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.003350:0:55719:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515e00/0x8e7c99ee62ab16c6 lrc: 4/1,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b835cc expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.003377:0:55719:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515e00/0x8e7c99ee62ab16c6 lrc: 4/1,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b835cc expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.003389:0:55719:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515e00/0x8e7c99ee62ab16c6 lrc: 4/1,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b835cc expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.003404:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515e00/0x8e7c99ee62ab16c6 lrc: 5/1,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b835cc expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.003414:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515e00/0x8e7c99ee62ab16c6 lrc: 5/1,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b835cc expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.003428:0:55719:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515e00/0x8e7c99ee62ab16c6 lrc: 4/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b835cc expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826425.003441:0:55719:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff976889efbc00 x1874670563986816/t8589935217(8589935217) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826440 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787826425.003456:0:55719:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efbc00 x1874670563986816/t8589935217(8589935217) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826440 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826425.003473:0:55719:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515e00/0x8e7c99ee62ab16c6 lrc: 3/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b835cc expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.003489:0:55719:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 8 res 8589935621/29 (0 0) 00000080:00200000:2.0:1787826425.003496:0:55719:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1787826425.003566:0:55719:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab16c6 00000080:00010000:2.0:1787826425.003575:0:55719:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab16c6 from it ffffb2d44225bbf0 00010000:00010000:2.0:1787826425.003579:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515e00/0x8e7c99ee62ab16c6 lrc: 3/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b835cc expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.003587:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515e00/0x8e7c99ee62ab16c6 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b835cc expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:2.0:1787826425.003623:0:55719:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efbc00 x1874670563986816/t8589935217(8589935217) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826440 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:2.0:1787826425.003727:0:55719:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=800466af arg=7ffe1792782c 00000080:00200000:2.0:1787826425.003904:0:55719:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:2.0:1787826425.003925:0:55719:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826425.003939:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826425.003951:0:55719:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826425.003958:0:55719:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826425.003964:0:55719:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:2.0:1787826425.003968:0:55719:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:2.0:1787826425.003972:0:55719:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44225bbf8 00010000:00010000:2.0:1787826425.003975:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826425.003983:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826425.003993:0:55719:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:2.0:1787826425.004001:0:55719:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=24 00000080:00200000:2.0:1787826425.004007:0:55719:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787826425.004023:0:55719:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 3 00000080:00200000:2.0:1787826425.004029:0:55719:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=. :VOLATILE:0000:7BD4BC4A:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), file ffff97688a0c2d80, open_flags 2f9c1, mode 8180 opened 0 00000080:00200000:2.0:1787826425.004036:0:55719:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=. :VOLATILE:0000:7BD4BC4A:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=open|creat 00000002:00010000:2.0:1787826425.004059:0:55719:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: . :VOLATILE:0000:7BD4BC4A:fd=03,[0x200000405:0x40:0x0]) in obj [0x200000bd0:0x4:0x0], intent: open|creat flags 0574702 00010000:00010000:2.0:1787826425.004124:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514600/0x8e7c99ee62ab16cd lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.004132:0:55719:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514600/0x8e7c99ee62ab16cd lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1787826425.004145:0:55719:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1787826425.004152:0:55719:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889ef9a40 x1874670563987200/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/66640 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1787826425.004162:0:55719:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514600/0x8e7c99ee62ab16cd lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826425.004196:0:55719:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563987200:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826425.004369:0:55719:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35240 going to sleep for 11 seconds 00000100:00100000:2.0:1787826425.008318:0:55719:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563987200:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826425.008353:0:55719:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514600/0x8e7c99ee62ab16cd lrc: 4/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b835e1 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.008403:0:55719:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514600/0x8e7c99ee62ab16cd lrc: 4/0,1 mode: --/PR res: [0x200000405:0x40:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b835e1 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.008412:0:55719:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514600/0x8e7c99ee62ab16cd lrc: 4/0,1 mode: --/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b835e1 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.008426:0:55719:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514600/0x8e7c99ee62ab16cd lrc: 4/0,1 mode: PR/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b835e1 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.008436:0:55719:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514600/0x8e7c99ee62ab16cd lrc: 4/0,1 mode: PR/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b835e1 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.008451:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514600/0x8e7c99ee62ab16cd lrc: 5/1,1 mode: PR/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b835e1 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.008467:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514600/0x8e7c99ee62ab16cd lrc: 5/1,1 mode: PR/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b835e1 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.008475:0:55719:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514600/0x8e7c99ee62ab16cd lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b835e1 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826425.008487:0:55719:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff976889ef9a40 x1874670563987200/t8589935218(8589935218) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787826425.008501:0:55719:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889ef9a40 x1874670563987200/t8589935218(8589935218) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826425.008513:0:55719:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514600/0x8e7c99ee62ab16cd lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b835e1 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.008523:0:55719:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/64 (0 0) 00000080:00200000:2.0:1787826425.008530:0:55719:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd0:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787826425.008554:0:55719:0:(namei.c:1628:ll_create_it()) VFS Op:name=. :VOLATILE:0000:7BD4BC4A:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=open|creat 00000080:00200000:2.0:1787826425.008767:0:55719:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000405:0x40:0x0](ffff976890b3db88) 00000080:00200000:2.0:1787826425.008775:0:55719:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x40:0x0], updating i_size 0 00000080:00200000:2.0:1787826425.008818:0:55719:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x40:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1787826425.008866:0:55719:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000405:0x40:0x0](ffff976890b3db88) 00000080:00010000:2.0:1787826425.008871:0:55719:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x40:0x0](ffff976890b3db88) for lock 0x8e7c99ee62ab16cd 00000080:00010000:2.0:1787826425.008883:0:55719:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x40:0x0](ffff976890b3db88) 00000080:00200008:2.0:1787826425.008901:0:55719:0:(file.c:1090:ll_file_open()) START file . :VOLATILE:0000:7BD4BC4A:fd=03:[0x200000405:0x40:0x0](ffff976890b3db88), flags 574701 00000002:00100000:2.0:1787826425.008926:0:55719:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889ef9a40 x1874670563987200/t8589935218(8589935218) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1787826425.008974:0:55719:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab16cd from it ffff9768876c9a20 00010000:00010000:2.0:1787826425.008979:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514600/0x8e7c99ee62ab16cd lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b835e1 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.008989:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514600/0x8e7c99ee62ab16cd lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b835e1 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826425.009014:0:55719:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x40:0x0](ffff976890b3db88), inode mode 8180 mask 40 00000080:00200000:2.0:1787826425.009083:0:55719:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x40:0x0](ffff976890b3db88), inode mode 8180 mask 2 00000080:00200000:2.0:1787826425.009095:0:55719:0:(xattr.c:414:ll_xattr_set()) VFS Op:inode=[0x200000405:0x40:0x0](ffff976890b3db88), xattr lov 00000002:00010000:2.0:1787826425.009119:0:55719:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x40:0x0]) in obj [0x200000405:0x40:0x0], intent: open flags 040000002 00010000:00010000:2.0:1787826425.009180:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16d4 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x40:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.009197:0:55719:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16d4 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x40:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1787826425.009210:0:55719:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1787826425.009218:0:55719:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efa1c0 x1874670563987456/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1787826425.009228:0:55719:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16d4 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x40:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826425.009249:0:55719:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa1c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563987456:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826425.009679:0:55719:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35240 going to sleep for 11 seconds 00000100:00100000:0.0:1787826425.013604:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826425.014057:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585287040 00000100:00100000:1.0:1787826425.014069:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585287040 00000100:00100000:0.0:1787826425.014303:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 575 00000100:00100000:0.0:1787826425.014315:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aef0c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6520:x1874670585287040:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826425.014339:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514600/0x8e7c99ee62ab16cd lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b835e1 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826425.014803:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826425.014812:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aef0c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6520:x1874670585287040:12345-192.168.203.139@tcp:104: Request processed in 499us (1220us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826425.014820:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 575 00010000:00010000:3.0:1787826425.014990:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826425.014999:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514600/0x8e7c99ee62ab16cd lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b835e1 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.015010:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514600 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826425.015016:0:42221:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514600/0x8e7c99ee62ab16cd lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b835e1 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.015025:0:42221:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514600/0x8e7c99ee62ab16cd lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b835e1 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.015034:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514600/0x8e7c99ee62ab16cd lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b835e1 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826425.015064:0:42221:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x40:0x0]: losing layout lock 00010000:00010000:3.0:1787826425.015122:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514600/0x8e7c99ee62ab16cd lrc: 2/0,0 mode: --/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b835e1 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.015130:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1787826425.015199:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563987584:192.168.203.139@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1787826425.015292:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514600/0x8e7c99ee62ab16cd lrc: 1/0,0 mode: --/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b835e1 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.015301:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514600/0x8e7c99ee62ab16cd lrc: 0/0,0 mode: --/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b835e1 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.015318:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826425.015320:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826425.015322:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826425.018272:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563987584:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826425.021688:0:55719:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa1c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563987456:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826425.021719:0:55719:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16d4 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x40:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.021736:0:55719:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16d4 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x40:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.021743:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16d4 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x40:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.021751:0:55719:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16d4 lrc: 3/0,0 mode: --/CR res: [0x200000405:0x40:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.021761:0:55719:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16d4 lrc: 4/0,0 mode: --/CR res: [0x200000405:0x40:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.021769:0:55719:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514e00 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:2.0:1787826425.021774:0:55719:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16d4 lrc: 4/0,0 mode: --/CR res: [0x200000405:0x40:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.021782:0:55719:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16d4 lrc: 4/0,0 mode: --/CR res: [0x200000405:0x40:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.021790:0:55719:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16d4 lrc: 5/0,0 mode: --/CR res: [0x200000405:0x40:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.021797:0:55719:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1787826425.021805:0:55719:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16d4 lrc: 3/0,0 mode: --/CR res: [0x200000405:0x40:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.021812:0:55719:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16d4 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x40:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826425.021924:0:55719:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x800002b, status=0 req@ffff976889efa1c0 x1874670563987456/t8589935219(8589935219) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787826425.021938:0:55719:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efa1c0 x1874670563987456/t8589935219(8589935219) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1787826425.021957:0:55719:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x40:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787826425.022012:0:55719:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x40:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787826425.022023:0:55719:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x40:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787826425.022073:0:55719:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x40:0x0], updating i_size 0 00000002:00100000:2.0:1787826425.022096:0:55719:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efa1c0 x1874670563987456/t8589935219(8589935219) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787826425.022141:0:55719:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889efa1c0 x1874670563987456/t8589935219(8589935219) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787826425.022153:0:55719:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787826425.022159:0:55719:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889ef8b40 x1874670563987840/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787826425.022196:0:55719:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8b40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563987840:192.168.203.139@tcp:35:lfs.0 00000100:00100000:2.0:1787826425.022342:0:55719:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35300 going to sleep for 11 seconds 00000100:00100000:2.0:1787826425.025042:0:55719:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8b40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563987840:192.168.203.139@tcp:35:lfs.0 00000002:00100000:2.0:1787826425.025087:0:55719:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889efa1c0 x1874670563987456/t8589935219(8589935219) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826441 ref 2 fl Complete:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1787826425.025134:0:55719:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787826425.025143:0:55719:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:2.0:1787826425.025163:0:55719:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826425.025179:0:55719:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 30 res 8589935621/29 (0 0) 00010000:00010000:2.0:1787826425.025287:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.025297:0:55719:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.025307:0:55719:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826425.025328:0:55719:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb480 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563987968:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826425.025709:0:55719:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35300 going to sleep for 11 seconds 00000100:00100000:1.0:1787826425.030162:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826425.030274:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585287168 00000100:00100000:0.0:1787826425.030287:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585287168 00000100:00100000:0.0:1787826425.030499:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 576 00000100:00100000:0.0:1787826425.030512:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0162580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:32404:x1874670585287168:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826425.030544:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515e00/0x8e7c99ee62ab16c6 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x22153cd396b835cc expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826425.030736:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826425.030743:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0162580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:32404:x1874670585287168:12345-192.168.203.139@tcp:104: Request processed in 234us (596us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826425.030752:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 576 00010000:00010000:2.0:1787826425.032225:0:49796:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826425.032242:0:49796:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515e00/0x8e7c99ee62ab16c6 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x22153cd396b835cc expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.032257:0:49796:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515e00 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:2.0:1787826425.032264:0:49796:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515e00/0x8e7c99ee62ab16c6 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x22153cd396b835cc expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.032274:0:49796:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515e00/0x8e7c99ee62ab16c6 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x22153cd396b835cc expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.032284:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515e00/0x8e7c99ee62ab16c6 lrc: 3/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x22153cd396b835cc expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.032322:0:49796:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab16b8 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b835b0 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826425.032739:0:49796:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515e00/0x8e7c99ee62ab16c6 lrc: 2/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x22153cd396b835cc expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.032747:0:49796:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1787826425.032841:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563988096:192.168.203.139@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1787826425.033476:0:49796:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515e00/0x8e7c99ee62ab16c6 lrc: 1/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x22153cd396b835cc expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.033489:0:49796:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515e00/0x8e7c99ee62ab16c6 lrc: 0/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x22153cd396b835cc expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.033507:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826425.033509:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826425.033512:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826425.037034:0:55719:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb480 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563987968:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826425.037069:0:55719:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.037083:0:55719:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.037098:0:55719:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.037111:0:55719:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.037126:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.037135:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.037143:0:55719:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826425.037156:0:55719:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff976889efb480 x1874670563987968/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 576/720 e 0 to 0 dl 1787826441 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826425.037169:0:55719:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787826425.037183:0:55719:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.037210:0:55719:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/29 (0 0) 00000080:00200000:2.0:1787826425.037218:0:55719:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787826425.037282:0:55719:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 35 -> 35 00000080:00010000:2.0:1787826425.037289:0:55719:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab16db 00000080:00010000:2.0:1787826425.037297:0:55719:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:2.0:1787826425.037299:0:55719:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab16db from it ffffb2d44225bcd8 00010000:00010000:2.0:1787826425.037302:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.037311:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826425.037354:0:55719:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 35 restore needed 0 00000001:00010000:2.0:1787826425.037383:0:55719:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:2.0:1787826425.037406:0:55719:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787826425.037410:0:55719:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff97688757cdb0, osc lock ffff976883f7a9c0, flags 0x400001002 00010000:00010000:2.0:1787826425.037432:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826425.037447:0:55719:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826425.037457:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826425.037465:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826425.037509:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab16e2 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.037518:0:55719:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab16e2 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1787826425.037680:0:55719:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:2.0:1787826425.037703:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563988224:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826425.040980:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563988096:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826425.041172:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826425.041279:0:42210:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:0.0:1787826425.041299:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585287296 00000100:00100000:0.0:1787826425.041307:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585287296 00000100:00100000:0.0:1787826425.041383:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 577 00000100:00100000:0.0:1787826425.041395:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768895b9c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30694:x1874670585287296:12345-192.168.203.139@tcp:106: 00010000:00010000:0.0:1787826425.041415:0:42210:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826425.041431:0:42210:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00000020:00010000:0.0:1787826425.041462:0:42210:0:(cl_object.c:322:cl_object_glimpse()) header@000000000e631c06[0x110, 1, [0x200000405:0x3e:0x0] hash] 00000020:00010000:0.0:1787826425.041465:0:42210:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826424 blocks: 0 00000020:00010000:0.0:1787826425.041469:0:42210:0:(cl_object.c:322:cl_object_glimpse()) header@000000000964cd23[0x116, 2, [0x280000401:0x10d:0x0]] 00000020:00010000:0.0:1787826425.041470:0:42210:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826424 blocks: 0 00000100:00100000:0.0:1787826425.041532:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768895b9c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30694:x1874670585287296:12345-192.168.203.139@tcp:106: Request processed in 139us (360us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826425.041552:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 577 00010000:00010000:2.0:1787826425.044754:0:2403:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab16e2 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.044784:0:2403:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab16e2 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.044800:0:2403:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab16e2 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.044809:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab16e2 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.044818:0:2403:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab16e2 lrc: 5/1,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787826425.044872:0:2403:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:2.0:1787826425.044880:0:2403:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1787826425.044950:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab16e2 lrc: 4/1,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.044960:0:2403:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab16e2 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.044969:0:2403:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab16e2 lrc: 4/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.044978:0:2403:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514e00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:2.0:1787826425.044986:0:2403:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab16e2 lrc: 5/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.044999:0:2403:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1787826425.045008:0:2403:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab16e2 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.045017:0:2403:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab16e2 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826425.045040:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563988224:192.168.203.139@tcp:101:lfs.0 00000080:00200000:0.0:1787826425.045105:0:55719:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787826425.045112:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf3b40 (ffff976890d10ea8/0) at cl_lock_release():245 00000020:00010000:0.0:1787826425.045116:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf3b40 (ffff976890d10ea8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826425.045123:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757cdb0 (ffff976890d106c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826425.045128:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf3b40 (ffff976890d10ea8/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826425.045131:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757cdb0 (ffff976890d10ea8/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826425.045147:0:55719:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 35 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787826425.045178:0:55719:0:(file.c:6313:ll_getattr_dentry()) START file . :VOLATILE:0000:7BD4BC4A:fd=03:[0x200000405:0x40:0x0](ffff976890b3db88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826425.045197:0:55719:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x40:0x0](ffff976890b3db88),name=. :VOLATILE:0000:7BD4BC4A:fd=03 00000002:00010000:0.0:1787826425.045227:0:55719:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x40:0x0]) in obj [0x200000405:0x40:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826425.045299:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab16e9 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x40:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.045314:0:55719:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab16e9 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x40:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.045326:0:55719:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab16e9 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x40:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826425.045353:0:55719:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aecf00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563988352:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826425.045757:0:55719:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e900 going to sleep for 11 seconds 00000100:00100000:0.0:1787826425.048933:0:55719:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aecf00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563988352:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826425.048966:0:55719:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab16e9 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x40:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83620 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.048980:0:55719:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab16e9 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83620 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.048994:0:55719:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab16e9 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83620 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.049007:0:55719:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab16e9 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83620 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.049021:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab16e9 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83620 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.049029:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab16e9 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83620 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.049042:0:55719:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab16e9 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83620 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826425.049055:0:55719:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff976882aecf00 x1874670563988352/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 576/720 e 0 to 0 dl 1787826441 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826425.049069:0:55719:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab16e9 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83620 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826425.049085:0:55719:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab16e9 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83620 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.049099:0:55719:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/64 (0 0) 00000080:00200000:0.0:1787826425.049107:0:55719:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000405:0x40:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826425.049177:0:55719:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x40:0x0]: layout version change: 4294967294 -> 1 00000080:00010000:0.0:1787826425.049183:0:55719:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x40:0x0](ffff976890b3db88) for lock 0x8e7c99ee62ab16e9 00000080:00010000:0.0:1787826425.049201:0:55719:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x40:0x0](ffff976890b3db88) 00000080:00010000:0.0:1787826425.049204:0:55719:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab16e9 from it ffffb2d44225bcd8 00010000:00010000:0.0:1787826425.049208:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab16e9 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83620 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.049219:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab16e9 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83620 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826425.049262:0:55719:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x40:0x0] ignore/verify layout 0/1, layout version 35 restore needed 0 00000001:00010000:0.0:1787826425.049275:0:55719:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x40:0x0] 00000008:00010000:0.0:1787826425.049298:0:55719:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826425.049303:0:55719:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e55930, osc lock ffff9768849a34b8, flags 0x400001002 00010000:00010000:0.0:1787826425.049482:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab16f0 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.049496:0:55719:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab16f0 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826425.049713:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aecf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563988480:192.168.203.139@tcp:101:lfs.0 00000020:00200000:0.0:1787826425.049862:0:55719:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787826425.056893:0:2402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab16f0 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.056932:0:2402:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab16f0 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83627 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.056951:0:2402:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab16f0 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83627 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.056965:0:2402:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab16f0 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83627 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826425.056982:0:2402:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab16f0 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83627 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826425.056996:0:2402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1787826425.057017:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab16f0 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83627 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.057033:0:2402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab16f0 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83627 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.057040:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab16f0 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83627 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.057050:0:2402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab16f0 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83627 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1787826425.057078:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aecf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563988480:192.168.203.139@tcp:101:lfs.0 00000080:00200000:2.0:1787826425.057201:0:55719:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x40:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1787826425.057210:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf3b40 (ffff976890d10ea8/0) at cl_lock_release():245 00000020:00010000:2.0:1787826425.057214:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf3b40 (ffff976890d10ea8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787826425.057221:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e55930 (ffff976890d106c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787826425.057226:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf3b40 (ffff976890d10ea8/0) at cl_lock_fini():72 00000020:00010000:2.0:1787826425.057230:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e55930 (ffff976890d10ea8/1) at cl_lock_fini():72 00000080:00200000:2.0:1787826425.057248:0:55719:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x40:0x0] ignore/verify layout 0/1, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200080:2.0:1787826425.057280:0:55719:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=7ffe17929974 00000002:00010000:2.0:1787826425.057320:0:55719:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 01004040000001 00010000:00010000:2.0:1787826425.057402:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16f7 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.057411:0:55719:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x81000000001000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16f7 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1787826425.057424:0:55719:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1787826425.057434:0:55719:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efb480 x1874670563988736/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1787826425.057445:0:55719:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16f7 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x81000000000000 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826425.057470:0:55719:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb480 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563988736:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826425.057877:0:55719:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35c00 going to sleep for 11 seconds 00000100:00100000:2.0:1787826425.075416:0:55719:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb480 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563988736:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826425.075449:0:55719:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16f7 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83635 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.075463:0:55719:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16f7 lrc: 4/1,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83635 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.075478:0:55719:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16f7 lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83635 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.075492:0:55719:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16f7 lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83635 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.075516:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16f7 lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83635 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.075526:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16f7 lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83635 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.075534:0:55719:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16f7 lrc: 4/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83635 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826425.075552:0:55719:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x600002b, status=0 req@ffff976889efb480 x1874670563988736/t8589935221(8589935221) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787826425.075797:0:55719:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efb480 x1874670563988736/t8589935221(8589935221) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826425.075815:0:55719:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16f7 lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83635 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.075859:0:55719:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 2 res 8589935621/29 (0 0) 00000080:00200000:2.0:1787826425.075866:0:55719:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000002:00100000:2.0:1787826425.075891:0:55719:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efb480 x1874670563988736/t8589935221(8589935221) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1787826425.075901:0:55719:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab16f7 00000080:00010000:2.0:1787826425.075908:0:55719:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab16f7 from it ffffb2d44225bcc0 00010000:00010000:2.0:1787826425.075912:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16f7 lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83635 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.075920:0:55719:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16f7 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83635 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200008:2.0:1787826425.075998:0:55719:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787826425.076007:0:55719:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:2.0:1787826425.076023:0:55719:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826425.076033:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.076051:0:55719:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826425.076058:0:55719:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826425.076063:0:55719:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:2.0:1787826425.076067:0:55719:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab16db 00000080:00010000:2.0:1787826425.076071:0:55719:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab16db from it ffffb2d44225bcd8 00010000:00010000:2.0:1787826425.076074:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.076083:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826425.076116:0:55719:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 1 restore needed 0 00000001:00010000:2.0:1787826425.076130:0:55719:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:2.0:1787826425.076154:0:55719:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787826425.076158:0:55719:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff97688757cfb0, osc lock ffff976883f7b5f0, flags 0x400001002 00010000:00010000:2.0:1787826425.076177:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826425.076199:0:55719:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826425.076209:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826425.076224:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826425.076264:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab16fe lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.076272:0:55719:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab16fe lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1787826425.076325:0:55719:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:2.0:1787826425.076343:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1787826425.076376:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563988864:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826425.079039:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826425.079737:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585287424 00000100:00100000:0.0:1787826425.079751:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585287424 00000100:00100000:1.0:1787826425.079863:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 578 00000100:00100000:1.0:1787826425.079966:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0163c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:8436:x1874670585287424:12345-192.168.203.139@tcp:106: 00010000:00010000:1.0:1787826425.079987:0:42210:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1787826425.080003:0:42210:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00000020:00010000:1.0:1787826425.080038:0:42210:0:(cl_object.c:322:cl_object_glimpse()) header@000000000e631c06[0x110, 1, [0x200000405:0x3e:0x0] hash] 00000020:00010000:1.0:1787826425.080040:0:42210:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826424 blocks: 0 00000020:00010000:1.0:1787826425.080044:0:42210:0:(cl_object.c:322:cl_object_glimpse()) header@000000000964cd23[0x116, 2, [0x280000401:0x10d:0x0]] 00000020:00010000:1.0:1787826425.080045:0:42210:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826424 blocks: 0 00000100:00100000:1.0:1787826425.080123:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0163c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:8436:x1874670585287424:12345-192.168.203.139@tcp:106: Request processed in 163us (1099us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826425.086780:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 578 00010000:00010000:2.0:1787826425.089811:0:2403:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab16fe lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.090241:0:2403:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab16fe lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.090258:0:2403:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab16fe lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.090267:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab16fe lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.090283:0:2403:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab16fe lrc: 5/1,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787826425.090301:0:2403:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:2.0:1787826425.090310:0:2403:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1787826425.090402:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab16fe lrc: 4/1,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.090411:0:2403:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab16fe lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.090420:0:2403:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab16fe lrc: 4/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.090429:0:2403:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515e00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:2.0:1787826425.090434:0:2403:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab16fe lrc: 5/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.090448:0:2403:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1787826425.090457:0:2403:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab16fe lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.090464:0:2403:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab16fe lrc: 0/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826425.090502:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563988864:192.168.203.139@tcp:101:lfs.0 00000080:00200000:0.0:1787826425.091916:0:55719:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787826425.091930:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf3b40 (ffff976890d10ea8/0) at cl_lock_release():245 00000020:00010000:0.0:1787826425.091935:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf3b40 (ffff976890d10ea8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826425.091942:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757cfb0 (ffff976890d106c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826425.091946:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf3b40 (ffff976890d10ea8/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826425.091949:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757cfb0 (ffff976890d10ea8/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826425.091970:0:55719:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 35 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787826425.092001:0:55719:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826425.092011:0:55719:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:0.0:1787826425.092041:0:55719:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826425.092065:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826425.092088:0:55719:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826425.092097:0:55719:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826425.092106:0:55719:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:0.0:1787826425.092110:0:55719:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab16db 00000080:00010000:0.0:1787826425.092117:0:55719:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab16db from it ffffb2d44225bcd8 00010000:00010000:0.0:1787826425.092121:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826425.092129:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826425.092168:0:55719:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 35 restore needed 0 00000001:00010000:0.0:1787826425.092177:0:55719:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:0.0:1787826425.092209:0:55719:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826425.092213:0:55719:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e553b0, osc lock ffff9768849a3248, flags 0x400001002 00010000:00010000:0.0:1787826425.092232:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826425.092242:0:55719:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826425.092251:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826425.092265:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826425.092312:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab1705 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.092320:0:55719:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab1705 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826425.092368:0:55719:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1787826425.092384:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1787826425.092407:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aecf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563988992:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826425.103664:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826425.104273:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585287552 00000100:00100000:1.0:1787826425.104287:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585287552 00000100:00100000:1.0:1787826425.104361:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 579 00000100:00100000:1.0:1787826425.104373:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0160f00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8545:x1874670585287552:12345-192.168.203.139@tcp:106: 00010000:00010000:1.0:1787826425.104464:0:53061:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1787826425.104569:0:53061:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00000020:00010000:1.0:1787826425.104605:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@000000000e631c06[0x110, 1, [0x200000405:0x3e:0x0] hash] 00000020:00010000:1.0:1787826425.104608:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826424 blocks: 0 00000020:00010000:1.0:1787826425.104612:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@000000000964cd23[0x116, 2, [0x280000401:0x10d:0x0]] 00000020:00010000:1.0:1787826425.104614:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826424 blocks: 0 00000100:00100000:1.0:1787826425.104691:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0160f00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8545:x1874670585287552:12345-192.168.203.139@tcp:106: Request processed in 320us (1041us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826425.104699:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 579 00010000:00010000:0.0:1787826425.108715:0:2402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab1705 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.108746:0:2402:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab1705 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.108762:0:2402:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab1705 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.108772:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab1705 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.108786:0:2402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab1705 lrc: 5/1,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826425.108801:0:2402:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:0.0:1787826425.108811:0:2402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1787826425.108941:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab1705 lrc: 4/1,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.108950:0:2402:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab1705 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.108960:0:2402:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab1705 lrc: 4/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.108968:0:2402:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925da00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:0.0:1787826425.108975:0:2402:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab1705 lrc: 5/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.108989:0:2402:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787826425.108997:0:2402:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab1705 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.109006:0:2402:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000002bd33d74/0x8e7c99ee62ab1705 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826425.109037:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aecf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563988992:192.168.203.139@tcp:101:lfs.0 00000080:00200000:2.0:1787826425.113897:0:55719:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1787826425.113912:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf3b40 (ffff976890d10ea8/0) at cl_lock_release():245 00000020:00010000:2.0:1787826425.113918:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf3b40 (ffff976890d10ea8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787826425.113925:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e553b0 (ffff976890d106c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787826425.113931:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf3b40 (ffff976890d10ea8/0) at cl_lock_fini():72 00000020:00010000:2.0:1787826425.113934:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e553b0 (ffff976890d10ea8/1) at cl_lock_fini():72 00000080:00200000:2.0:1787826425.113958:0:55719:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 35 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200080:2.0:1787826425.113988:0:55719:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=801066da arg=7ffe17929870 00000080:00200000:2.0:1787826425.114001:0:55719:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 35 restore needed 0 00020000:00200000:2.0:1787826425.114014:0:55719:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1787826425.114799:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563989120:192.168.203.139@tcp:1:lfs.0 00000100:00100000:1.0:1787826425.128167:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826425.128281:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585287680 00000100:00100000:0.0:1787826425.128293:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585287680 00000100:00100000:0.0:1787826425.128364:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 580 00000100:00100000:0.0:1787826425.128376:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0163480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30696:x1874670585287680:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826425.128399:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826425.128497:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826425.128504:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0163480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30696:x1874670585287680:12345-192.168.203.139@tcp:104: Request processed in 131us (353us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826425.128511:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 580 00010000:00010000:3.0:1787826425.128594:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826425.128602:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826425.128612:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976888fb9800 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826425.128619:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400020000 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1787826425.128648:0:42220:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x3e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787826425.128676:0:42220:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x3e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787826425.128733:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 2/0,0 mode: --/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826425.128742:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826425.128778:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 1/0,0 mode: --/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826425.128787:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000288028c5/0x8e7c99ee62ab165d lrc: 0/0,0 mode: --/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x22153cd396b8347c expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826425.128814:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826425.128816:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826425.128817:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826425.128881:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826425.128898:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563989248:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826425.139575:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563989248:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826425.139624:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563989120:192.168.203.139@tcp:1:lfs.0 00000080:00200000:2.0:1787826425.139940:0:55719:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:2.0:1787826425.139954:0:55719:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:2.0:1787826425.139992:0:55719:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787826425.140002:0:55719:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:2.0:1787826425.140032:0:55719:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826425.140056:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826425.140072:0:55719:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787826425.140082:0:55719:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826425.140090:0:55719:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:2.0:1787826425.140095:0:55719:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab16db 00000080:00010000:2.0:1787826425.140105:0:55719:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab16db from it ffffb2d44225bcd8 00010000:00010000:2.0:1787826425.140110:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826425.140119:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1787826425.140150:0:55719:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 35 restore needed 0 00000001:00010000:2.0:1787826425.140163:0:55719:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:2.0:1787826425.140198:0:55719:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787826425.140203:0:55719:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff97688757c6b0, osc lock ffff976883f7a888, flags 0x400001002 00010000:00010000:2.0:1787826425.140267:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.140278:0:55719:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1787826425.140328:0:55719:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:2.0:1787826425.140343:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1787826425.140362:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563989376:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826425.158410:0:2404:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.158453:0:2404:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.158473:0:2404:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.158492:0:2404:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787826425.158662:0:2404:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1787826425.158677:0:2404:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1787826425.158699:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.158709:0:2404:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.158717:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.158727:0:2404:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1787826425.158754:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563989376:192.168.203.139@tcp:101:lfs.0 00000080:00200000:0.0:1787826425.158969:0:55719:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787826425.158978:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf3b40 (ffff976890d10ea8/0) at cl_lock_release():245 00000020:00010000:0.0:1787826425.158981:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf3b40 (ffff976890d10ea8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826425.158987:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757c6b0 (ffff976890d106c8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826425.158992:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf3b40 (ffff976890d10ea8/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826425.158995:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757c6b0 (ffff976890d10ea8/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826425.159014:0:55719:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 35 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1787826425.219415:0:55719:0:(file.c:5224:ll_file_seek()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1787826425.219442:0:55719:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787826425.219456:0:55719:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:2.0:1787826425.219487:0:55719:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787826425.219492:0:55719:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff97688757cbb0, osc lock ffff976883f7a750, flags 0x400001002 00010000:00010000:2.0:1787826425.219527:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.219550:0:55719:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1787826425.219568:0:55719:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787826425.219574:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.219585:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1787826425.219603:0:55719:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787826425.219616:0:55719:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1787826425.219622:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf0000 (ffff976890d10d88/0) at cl_lock_release():245 00000020:00010000:2.0:1787826425.219626:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf0000 (ffff976890d10d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787826425.219629:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757cbb0 (ffff976890d105a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787826425.219633:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf0000 (ffff976890d10d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1787826425.219636:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757cbb0 (ffff976890d10d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1787826425.219654:0:55719:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 35 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1787826425.219663:0:55719:0:(file.c:5262:ll_file_seek()) COMPLETED file f60b.sanity-flr:[0x200000405:0x1d:0x0], offset: 0, type: (null), rc = 0 00000080:00200008:2.0:1787826425.219669:0:55719:0:(file.c:5224:ll_file_seek()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0], offset: 0, type: SEEK_HOLE 00000080:00200000:2.0:1787826425.219673:0:55719:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 35 restore needed 0 00020000:00200000:2.0:1787826425.219681:0:55719:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787826425.219703:0:55719:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000080:00200000:2.0:1787826425.219713:0:55719:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787826425.219720:0:55719:0:(file.c:1848:ll_io_set_mirror()) f60b.sanity-flr: desiginated mirror: 0 00000080:00200000:2.0:1787826425.219723:0:55719:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787826425.219728:0:55719:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787826425.219733:0:55719:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 18446744073709551615] 00000008:00010000:2.0:1787826425.219740:0:55719:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 8 00000008:00010000:2.0:1787826425.219742:0:55719:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff97688757cbb0, osc lock ffff976883f7a750, flags 0x0 00010000:00010000:2.0:1787826425.219751:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826425.219762:0:55719:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826425.219772:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826425.219781:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826425.219789:0:55719:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826425.219799:0:55719:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1787826425.219805:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976890fb78a8 (ffff976890d10d88/0) at cl_lock_release():245 00000020:00010000:2.0:1787826425.219808:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976890fb78a8 (ffff976890d10d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787826425.219810:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757cbb0 (ffff976890d105a8/1) at cl_lock_cancel():142 00010000:00010000:2.0:1787826425.219814:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 4/1,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826425.219858:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00010000:2.0:1787826425.219868:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976890fb78a8 (ffff976890d10d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1787826425.219871:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757cbb0 (ffff976890d10d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1787826425.219878:0:55719:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 35 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1787826425.219882:0:55719:0:(file.c:5262:ll_file_seek()) COMPLETED file f60b.sanity-flr:[0x200000405:0x1d:0x0], offset: 0, type: SEEK_HOLE, rc = -6 00000080:00200080:2.0:1787826425.219928:0:55719:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=66f4 arg=55a 00000080:00200008:2.0:1787826425.219949:0:55719:0:(file.c:2485:do_file_read_iter()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0], ppos: 0, count: 67108864 00000080:00200000:2.0:1787826425.219960:0:55719:0:(file.c:2090:ll_file_io_generic()) f60b.sanity-flr: read ppos: 0, bytes: 67108864 00000080:00200000:2.0:1787826425.219968:0:55719:0:(file.c:1848:ll_io_set_mirror()) f60b.sanity-flr: desiginated mirror: 0 00000020:00200000:2.0:1787826425.219975:0:55719:0:(cl_io.c:218:cl_io_rw_init()) header@00000000a0ca5d5e[0x110, 2, [0x200000405:0x1d:0x0] hash] 00000020:00200000:2.0:1787826425.219978:0:55719:0:(cl_io.c:218:cl_io_rw_init()) io range: 1 [0, 67108864) 2048 0 00000080:00200000:2.0:1787826425.219981:0:55719:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 35 restore needed 0 00000080:00200000:2.0:1787826425.219990:0:55719:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 16383] 00020000:00200000:2.0:1787826425.219998:0:55719:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 67108864) 67108864, 67108864 00020000:00200000:2.0:1787826425.220001:0:55719:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787826425.220011:0:55719:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 16383] 00000008:00010000:2.0:1787826425.220017:0:55719:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 11 00000008:00010000:2.0:1787826425.220020:0:55719:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff97688757cbb0, osc lock ffff976883f7a750, flags 0x40000 00000080:00200000:2.0:1787826425.220025:0:55719:0:(vvp_io.c:874:vvp_io_read_start()) f60b.sanity-flr: read [0, 67108864) 00000001:00010000:2.0:1787826425.220032:0:55719:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:2.0:1787826425.220037:0:55719:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787826425.220039:0:55719:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff97688757c4b0, osc lock ffff976883f7b4b8, flags 0x400001002 00010000:00010000:2.0:1787826425.220048:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826425.220058:0:55719:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00200000:2.0:1787826425.220068:0:55719:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787826425.220071:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826425.220079:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00000020:00200000:2.0:1787826425.220088:0:55719:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787826425.220092:0:55719:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1787826425.220094:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf0000 (ffff976890d10d88/0) at cl_lock_release():245 00000020:00010000:2.0:1787826425.220097:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf0000 (ffff976890d10d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787826425.220100:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757c4b0 (ffff976890d105a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787826425.220103:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf0000 (ffff976890d10d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1787826425.220105:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757c4b0 (ffff976890d10d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1787826425.220111:0:55719:0:(vvp_io.c:932:vvp_io_read_start()) count:67108864 iocb pos:0 00000080:00200000:2.0:1787826425.220119:0:55719:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), size=67108864 (max 335544320), offset=0=0x0, pages 16384 (max 81920), lockless, parallel 00000020:00010000:2.0:1787826425.220128:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976890fb78a8 (ffff976890d10d88/0) at cl_lock_release():245 00000020:00010000:2.0:1787826425.220131:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976890fb78a8 (ffff976890d10d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787826425.220134:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757cbb0 (ffff976890d105a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787826425.220137:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976890fb78a8 (ffff976890d10d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1787826425.220139:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757cbb0 (ffff976890d10d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1787826425.220144:0:55719:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000020:00200000:2.0:1787826425.220148:0:55719:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787826425.220151:0:55719:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 16383] 00000080:00200000:2.0:1787826425.220156:0:55719:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 35 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787826425.220160:0:55719:0:(file.c:2240:ll_file_io_generic()) f60b.sanity-flr: 1 io complete with rc: 0, result: 0, restart: 0 00000020:00200000:2.0:1787826425.220163:0:55719:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787826425.220168:0:55719:0:(file.c:2326:ll_file_io_generic()) iot: 1, result: 0 00000080:00200008:2.0:1787826425.220180:0:55719:0:(file.c:5401:ll_fsync()) START file: name=. :VOLATILE:0000:7BD4BC4A:fd=03, fid=[0x200000405:0x40:0x0], start=0, end=9223372036854775807, datasync=0, need_sync 1/0 00000100:00100000:2.0:1787826425.221129:0:55719:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563989504:192.168.203.139@tcp:44:lfs.0 00000100:00100000:2.0:1787826425.221257:0:55719:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35c00 going to sleep for 11 seconds 00000100:00100000:2.0:1787826425.226501:0:55719:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9768877a4800: committing for last_committed 8589935221 gen 1 00000100:00100000:2.0:1787826425.226576:0:55719:0:(client.c:3174:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff976882aed680 x1874670563986432/t8589935216(8589935216) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826440 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1787826425.226592:0:55719:0:(client.c:3174:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff976889efbc00 x1874670563986816/t8589935217(8589935217) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826440 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1787826425.226601:0:55719:0:(client.c:3174:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff976889ef9a40 x1874670563987200/t8589935218(8589935218) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826441 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1787826425.226674:0:55719:0:(client.c:3174:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff976889efb480 x1874670563988736/t8589935221(8589935221) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826441 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1787826425.226688:0:55719:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563989504:192.168.203.139@tcp:44:lfs.0 00000080:00200000:2.0:1787826425.226718:0:55719:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x40:0x0] ignore/verify layout 0/0, layout version 35 restore needed 0 00020000:00200000:2.0:1787826425.226736:0:55719:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1787826425.226996:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563989632:192.168.203.139@tcp:16:lfs.0 00000100:00100000:1.0:1787826425.240688:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826425.241078:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585287936 00000100:00100000:0.0:1787826425.241090:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585287936 00000100:00100000:0.0:1787826425.241175:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 581 00000100:00100000:0.0:1787826425.241197:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0161a40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:28853:x1874670585287936:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826425.241222:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000fdf26208/0x8e7c99ee62ab1664 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000030000 nid: local remote: 0x22153cd396b8348a expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826425.241332:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826425.241339:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0161a40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:28853:x1874670585287936:12345-192.168.203.139@tcp:104: Request processed in 154us (668us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826425.241347:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 581 00010000:00010000:3.0:1787826425.244012:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826425.244031:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000fdf26208/0x8e7c99ee62ab1664 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000030000 nid: local remote: 0x22153cd396b8348a expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.244050:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976890fdf600 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826425.244061:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000fdf26208/0x8e7c99ee62ab1664 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400030000 nid: local remote: 0x22153cd396b8348a expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.244154:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000fdf26208/0x8e7c99ee62ab1664 lrc: 2/0,0 mode: --/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400030020 nid: local remote: 0x22153cd396b8348a expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.244163:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826425.244213:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000fdf26208/0x8e7c99ee62ab1664 lrc: 1/0,0 mode: --/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400030020 nid: local remote: 0x22153cd396b8348a expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.244222:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000fdf26208/0x8e7c99ee62ab1664 lrc: 0/0,0 mode: --/PW res: [0x2c0000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400030020 nid: local remote: 0x22153cd396b8348a expref: -99 pid: 55703 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.244254:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826425.244256:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826425.244258:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826425.244275:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563989760:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826425.249933:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563989632:192.168.203.139@tcp:16:lfs.0 00000080:00200000:2.0:1787826425.250289:0:55719:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x40:0x0] updating i_size 0 i_blocks 0 00000080:00200000:2.0:1787826425.250309:0:55719:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x40:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1787826425.260965:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563989760:192.168.203.139@tcp:103:ldlm_bl.0 00000080:00200080:2.0:1787826425.273235:0:55719:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=801066da arg=7ffe17929870 00000080:00200000:2.0:1787826425.273271:0:55719:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787826425.273289:0:55719:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1787826425.273408:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563989888:192.168.203.139@tcp:1:lfs.0 00000100:00100000:3.0:1787826425.277736:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563989888:192.168.203.139@tcp:1:lfs.0 00000080:00200008:2.0:1787826425.278095:0:55719:0:(llite_lib.c:2427:ll_setattr_raw()) START file . :VOLATILE:0000:7BD4BC4A:fd=03:[0x200000405:0x40:0x0](ffff976890b3db88) current size 0, valid attrs 101f0, mode 9768, uid 0, gid 0, new size 4294967296, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826425.000000000 00010000:00010000:2.0:1787826425.278147:0:55719:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab16e9 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b83620 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826425.278197:0:55719:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x40:0x0]: losing layout lock 00010000:00010000:2.0:1787826425.278262:0:55719:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab16e9 lrc: 1/0,0 mode: --/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b83620 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.278280:0:55719:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826425.278285:0:55719:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab16e9 lrc: 0/0,0 mode: --/PR res: [0x200000405:0x40:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b83620 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1787826425.278316:0:55719:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787826425.278327:0:55719:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efa1c0 x1874670563990144/t0(0) o36->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826425.278357:0:55719:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa1c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563990144:192.168.203.139@tcp:36:lfs.0 00000100:00100000:0.0:1787826425.280388:0:55719:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35c00 going to sleep for 11 seconds 00000100:00100000:2.0:1787826425.285572:0:55719:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa1c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563990144:192.168.203.139@tcp:36:lfs.0 00000080:00200000:2.0:1787826425.285647:0:55719:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x40:0x0] ignore/verify layout 0/0, layout version 35 restore needed 0 00000080:00010000:2.0:1787826425.285683:0:55719:0:(file.c:7076:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000405:0x40:0x0](ffff976890b3db88) 00000002:00010000:2.0:1787826425.285698:0:55719:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x40:0x0]) in obj [0x200000405:0x40:0x0], intent: layout flags 0 00010000:00010000:2.0:1787826425.285779:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1713 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x40:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.285791:0:55719:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1713 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x40:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.285803:0:55719:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1713 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x40:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826425.287017:0:55719:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563990272:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826425.287244:0:55719:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35c00 going to sleep for 11 seconds 00000100:00100000:2.0:1787826425.298667:0:55719:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563990272:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826425.298709:0:55719:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1713 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x40:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83674 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.298725:0:55719:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1713 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x40:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83674 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826425.298742:0:55719:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff976889efb0c0 x1874670563990272/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 376/480 e 0 to 0 dl 1787826441 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826425.298755:0:55719:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1713 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x40:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83674 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826425.303148:0:55719:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1713 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x40:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83674 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.303178:0:55719:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/64 (0 0) 00000080:00200000:0.0:1787826425.303198:0:55719:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000405:0x40:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1787826425.303229:0:55719:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x40:0x0](ffff976890b3db88) for lock 0x8e7c99ee62ab1713 00000080:00010000:0.0:1787826425.303241:0:55719:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1713 from it ffffb2d44225ba78 00010000:00010000:0.0:1787826425.303246:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1713 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x40:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83674 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.303258:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1713 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x40:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83674 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826425.303306:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1713 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x40:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83674 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826425.303320:0:55719:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1713 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x40:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83674 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826425.303329:0:55719:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x40:0x0](0000000040abd9cb) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1713 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x40:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83674 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826425.303385:0:55719:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x40:0x0]: layout version change: 4294967294 -> 1 00010000:00010000:0.0:1787826425.303391:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1713 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x40:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83674 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826425.303399:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1713 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x40:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83674 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:0.0:1787826425.303426:0:55719:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826425.303445:0:55719:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:0.0:1787826425.303462:0:55719:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787826425.303467:0:55719:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e558b0, osc lock ffff9768849a2888, flags 0x0 00010000:00010000:0.0:1787826425.303490:0:55719:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9768ad0ae800 type 11 mode 2 res 11811161089/270 (0 18446744073709551615) 00010000:00010000:0.0:1787826425.303540:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.303550:0:55719:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.303561:0:55719:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826425.303593:0:55719:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aecf00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563990400:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826425.303697:0:55719:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741eb40 going to sleep for 11 seconds 00000100:00100000:0.0:1787826425.308802:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826425.308976:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585288064 00000100:00100000:1.0:1787826425.308989:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585288064 00000100:00100000:1.0:1787826425.309057:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 582 00000100:00100000:1.0:1787826425.309069:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aec3c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:14864:x1874670585288064:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826425.309199:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab16f0 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b83627 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826425.309300:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826425.309308:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aec3c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:14864:x1874670585288064:12345-192.168.203.139@tcp:104: Request processed in 241us (522us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826425.309317:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 582 00010000:00010000:2.0:1787826425.309585:0:49796:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826425.309595:0:49796:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab16f0 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b83627 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.309605:0:49796:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925ce00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:2.0:1787826425.309612:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab16f0 lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x22153cd396b83627 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826425.309668:0:49796:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x40:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787826425.309683:0:49796:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x40:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1787826425.309738:0:49796:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab16f0 lrc: 2/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b83627 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.309747:0:49796:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826425.309788:0:49796:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab16f0 lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b83627 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.309796:0:49796:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000ff4ed67a/0x8e7c99ee62ab16f0 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b83627 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826425.310858:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787826425.310864:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787826425.310866:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826425.311087:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563990528:192.168.203.139@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1787826425.311590:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826425.315031:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826425.315072:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585288192 00000100:00100000:1.0:1787826425.315093:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585288192 00000100:00100000:2.0:1787826425.315140:0:55719:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aecf00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563990400:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826425.315159:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 583 00000100:00100000:1.0:1787826425.315172:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0160b40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:6504:x1874670585288192:12345-192.168.203.139@tcp:105: 00010000:00010000:2.0:1787826425.315181:0:55719:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 4/0,1 mode: --/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.315207:0:55719:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 5/0,1 mode: --/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.315217:0:55719:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 5/0,1 mode: --/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.316754:0:42210:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 5/0,1 mode: --/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.316778:0:42210:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 5/0,1 mode: --/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.316791:0:42210:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 5/0,1 mode: --/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.316807:0:42210:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.316821:0:42210:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 6/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.316847:0:42210:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 6/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.316895:0:42210:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff97688925d000) 00010000:00010000:2.0:1787826425.316964:0:55719:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.316978:0:55719:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.317003:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787826425.317019:0:55719:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 4/0,2 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.317035:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.317045:0:55719:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826425.317263:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826425.317288:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563990656:192.168.203.139@tcp:2:lfs.0 00000100:00100000:1.0:1787826425.318409:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0160b40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:6504:x1874670585288192:12345-192.168.203.139@tcp:105: Request processed in 3229us (3382us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826425.318425:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 583 00000100:00100000:3.0:1787826425.324299:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563990528:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826425.332665:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563990656:192.168.203.139@tcp:2:lfs.0 00000020:00010000:2.0:1787826425.333212:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976890fb78a8 (ffff976890d10d88/0) at cl_lock_release():245 00000020:00010000:2.0:1787826425.333219:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976890fb78a8 (ffff976890d10d88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787826425.333225:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e558b0 (ffff976890d10ea8/1) at cl_lock_cancel():142 00010000:00010000:2.0:1787826425.333239:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.333258:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1787826425.333277:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976890fb78a8 (ffff976890d10d88/0) at cl_lock_fini():72 00000020:00010000:2.0:1787826425.333281:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e558b0 (ffff976890d10d88/1) at cl_lock_fini():72 00000080:00200000:2.0:1787826425.333306:0:55719:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x40:0x0] ignore/verify layout 0/0, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1787826425.333324:0:55719:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file . :VOLATILE:0000:7BD4BC4A:fd=03:[0x200000405:0x40:0x0](ffff976890b3db88) current size 0, valid attrs 100101f0, mode 9768, uid 0, gid 0, new size 4294967296, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826425.000000000, rc 0 00000080:00200008:2.0:1787826425.333482:0:55719:0:(llite_lib.c:2427:ll_setattr_raw()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8) current size 0, valid attrs 101f0, mode 9768, uid 0, gid 0, new size 0, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826425.000000000 00010000:00010000:2.0:1787826425.333525:0:55719:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826425.333553:0:55719:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab16b8 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b835b0 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826425.333611:0:55719:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 1/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826425.333619:0:55719:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826425.333622:0:55719:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab16db lrc: 0/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b8360b expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000020:00100000:2.0:1787826425.333648:0:55719:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787826425.333660:0:55719:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efb840 x1874670563990912/t0(0) o36->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826425.333687:0:55719:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563990912:192.168.203.139@tcp:36:lfs.0 00000100:00100000:2.0:1787826425.335214:0:55719:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35300 going to sleep for 11 seconds 00000100:00100000:2.0:1787826425.339666:0:55719:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563990912:192.168.203.139@tcp:36:lfs.0 00000080:00200000:2.0:1787826425.339731:0:55719:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 1 restore needed 0 00020000:00200000:2.0:1787826425.339754:0:55719:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787826425.339777:0:55719:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:2.0:1787826425.339797:0:55719:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1787826425.339804:0:55719:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff97688757c4b0, osc lock ffff976883f7a750, flags 0x0 00010000:00010000:2.0:1787826425.339873:0:55719:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976889533000 type 11 mode 2 res 10737419265/269 (0 18446744073709551615) 00010000:00010000:2.0:1787826425.339932:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.339946:0:55719:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.339959:0:55719:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826425.339984:0:55719:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef83c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563991040:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826425.340066:0:55719:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35300 going to sleep for 11 seconds 00000100:00100000:0.0:1787826425.345351:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826425.345494:0:42211:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1787826425.345519:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585288320 00000100:00100000:1.0:1787826425.345530:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585288320 00000100:00100000:1.0:1787826425.345637:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 584 00000100:00100000:1.0:1787826425.345648:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768895b8450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:8437:x1874670585288320:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826425.345672:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787826425.345770:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826425.345777:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768895b8450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:8437:x1874670585288320:12345-192.168.203.139@tcp:104: Request processed in 132us (440us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826425.345786:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 584 00010000:00010000:3.0:1787826425.345915:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826425.345925:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826425.345937:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515e00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826425.345943:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1787826425.345967:0:42221:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787826425.345984:0:42221:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787826425.346039:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 2/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826425.346047:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826425.346073:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 1/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826425.346083:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000004a622e6e/0x8e7c99ee62ab170c lrc: 0/0,0 mode: --/PR res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b83658 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826425.346101:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826425.346103:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826425.346105:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826425.346115:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787826425.346133:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563991168:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826425.352146:0:55719:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef83c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563991040:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826425.352201:0:55719:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 4/0,1 mode: --/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b83689 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.352220:0:55719:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 4/0,1 mode: --/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b83689 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.352232:0:55719:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 4/0,1 mode: --/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b83689 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826425.354709:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826425.355086:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585288448 00000100:00100000:1.0:1787826425.355098:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585288448 00000100:00100000:1.0:1787826425.355164:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 585 00000100:00100000:1.0:1787826425.355177:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aef0c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:10340:x1874670585288448:12345-192.168.203.139@tcp:105: 00010000:00010000:1.0:1787826425.355206:0:42210:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 5/0,1 mode: --/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b83689 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.355221:0:42210:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 5/0,1 mode: --/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b83689 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.355237:0:42210:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 5/0,1 mode: --/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b83689 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.355249:0:42210:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x22153cd396b83689 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.355259:0:42210:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 6/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x22153cd396b83689 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.355271:0:42210:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 6/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x22153cd396b83689 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.356136:0:42210:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff976883514800) 00000100:00100000:1.0:1787826425.356289:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aef0c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:10340:x1874670585288448:12345-192.168.203.139@tcp:105: Request processed in 1112us (1590us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826425.356300:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 585 00010000:00010000:2.0:1787826425.360350:0:55719:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b83689 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.360398:0:55719:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b83689 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.360419:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b83689 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787826425.360433:0:55719:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x22153cd396b83689 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.360448:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b83689 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.360457:0:55719:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b83689 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826425.361288:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1787826425.361319:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef83c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563991296:192.168.203.139@tcp:2:lfs.0 00000100:00100000:3.0:1787826425.367306:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563991168:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826425.367768:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef83c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563991296:192.168.203.139@tcp:2:lfs.0 00000020:00010000:0.0:1787826425.367867:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976890fb78a8 (ffff976890d10d88/0) at cl_lock_release():245 00000020:00010000:0.0:1787826425.367873:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976890fb78a8 (ffff976890d10d88/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826425.367881:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757c4b0 (ffff976890d10ea8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787826425.367895:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b83689 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.367915:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b83689 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787826425.367935:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976890fb78a8 (ffff976890d10d88/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826425.367938:0:55719:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757c4b0 (ffff976890d10d88/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826425.367970:0:55719:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 35 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787826425.367987:0:55719:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8) current size 0, valid attrs 100101f0, mode 9768, uid 0, gid 0, new size 0, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826425.000000000, rc 0 00000080:00200080:0.0:1787826425.368022:0:55719:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=b2a570 00010000:00010000:0.0:1787826425.368093:0:55719:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16f7 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0x22153cd396b83635 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.368113:0:55719:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514e00/0x8e7c99ee62ab16f7 lrc: 0/0,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x85809400000000 nid: local remote: 0x22153cd396b83635 expref: -99 pid: 55719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826425.368136:0:55719:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889efb480 x1874670563988736/t8589935221(8589935221) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826441 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826425.368148:0:55719:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826425.368154:0:55719:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aec3c0 x1874670563991552/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 504/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826425.368196:0:55719:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563991552:192.168.203.139@tcp:35:lfs.0 00000100:00100000:1.0:1787826425.370300:0:55719:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e780 going to sleep for 11 seconds 00000100:00100000:1.0:1787826425.377537:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826425.377597:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826425.377695:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585288576 00000100:00100000:0.0:1787826425.377707:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585288576 00000100:00100000:0.0:1787826425.377770:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585288704 00000100:00100000:0.0:1787826425.377773:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585288704 00000100:00100000:0.0:1787826425.377793:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 586 00000100:00100000:0.0:1787826425.377806:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0163c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+6:14257:x1874670585288576:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826425.377871:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab16b8 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b835b0 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826425.377975:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826425.377983:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0163c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+6:14257:x1874670585288576:12345-192.168.203.139@tcp:104: Request processed in 180us (471us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826425.377991:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 586 00000100:00100000:0.0:1787826425.378010:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 587 00000100:00100000:0.0:1787826425.378017:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0160f00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585288704:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826425.378027:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83563 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826425.378056:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:0.0:1787826425.378061:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0160f00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585288704:12345-192.168.203.139@tcp:104: Request processed in 45us (465us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826425.378067:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 587 00010000:00010000:2.0:1787826425.378299:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:2.0:1787826425.378310:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab16b8 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b835b0 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826425.378320:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925c600 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:2.0:1787826425.378326:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab16b8 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b835b0 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826425.378335:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab16b8 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b835b0 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826425.378345:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab16b8 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 3 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b835b0 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826425.378377:0:42220:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83563 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000002:00100000:2.0:1787826425.378442:0:42220:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976882aed680 x1874670563986432/t8589935216(8589935216) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826440 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787826425.378456:0:42220:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 2 is allocated opc 35, max 7 00000100:00100000:2.0:1787826425.378462:0:42220:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889ef8000 x1874670563991808/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787826425.378490:0:42220:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8000 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:42220:1874670563991808:192.168.203.139@tcp:35:ldlm_bl.0 00000100:00100000:2.0:1787826425.378563:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35300 going to sleep for 11 seconds 00010000:00010000:3.0:1787826425.381399:0:49796:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826425.381419:0:49796:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83563 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826425.381436:0:49796:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515200 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826425.381443:0:49796:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83563 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826425.381453:0:49796:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83563 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826425.381464:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b83563 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826425.381496:0:49796:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x1d:0x0]: losing layout lock 00010000:00010000:3.0:1787826425.381568:0:49796:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83563 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826425.381576:0:49796:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826425.381610:0:49796:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83563 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826425.381620:0:49796:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515200/0x8e7c99ee62ab16a3 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83563 expref: -99 pid: 55711 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826425.381639:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826425.381642:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826425.381645:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826425.381654:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787826425.381681:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563991936:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826425.385118:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563991936:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826425.385793:0:42220:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8000 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:42220:1874670563991808:192.168.203.139@tcp:35:ldlm_bl.0 00000002:00100000:2.0:1787826425.386206:0:42220:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976882aed680 x1874670563986432/t8589935216(8589935216) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826440 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1787826425.386301:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab16b8 lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b835b0 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826425.386311:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826425.386341:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab16b8 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b835b0 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826425.386350:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab16b8 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b835b0 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826425.386390:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826425.386393:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826425.386396:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826425.386413:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563992064:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826425.389364:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563992064:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826425.396348:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826425.396514:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585288832 00000100:00100000:1.0:1787826425.396537:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585288832 00000100:00100000:1.0:1787826425.396609:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 588 00000100:00100000:1.0:1787826425.396622:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aecf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585288832:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826425.396647:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1713 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x40:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83674 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826425.396755:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826425.396763:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aecf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585288832:12345-192.168.203.139@tcp:104: Request processed in 144us (431us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826425.396771:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 588 00010000:00010000:3.0:1787826425.396915:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826425.396925:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1713 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x40:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83674 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826425.396936:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514400 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826425.396942:0:42221:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1713 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x40:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83674 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826425.396953:0:42221:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1713 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x40:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83674 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826425.396962:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1713 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x40:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b83674 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826425.396993:0:42221:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x40:0x0]: losing layout lock 00010000:00010000:3.0:1787826425.397047:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1713 lrc: 2/0,0 mode: --/CR res: [0x200000405:0x40:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83674 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826425.397055:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826425.397090:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1713 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x40:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83674 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826425.397099:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1713 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x40:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83674 expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826425.397130:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826425.397133:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826425.397135:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826425.397202:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1787826425.397225:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563992192:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826425.402038:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563992192:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826425.408944:0:55719:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563991552:192.168.203.139@tcp:35:lfs.0 00000002:00100000:1.0:1787826425.408994:0:55719:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889efb480 x1874670563988736/t8589935221(8589935221) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826441 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1787826425.409048:0:55719:0:(file.c:7076:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000002:00010000:1.0:1787826425.409063:0:55719:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: layout flags 0 00010000:00010000:1.0:1787826425.409132:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.409144:0:55719:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.409154:0:55719:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55719 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826425.409175:0:55719:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0163480 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563992320:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826425.409282:0:55719:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3600 going to sleep for 11 seconds 00000100:00100000:1.0:1787826425.414882:0:55719:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0163480 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563992320:192.168.203.139@tcp:101:lfs.0 00010000:00010000:1.0:1787826425.414929:0:55719:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8369e expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.414952:0:55719:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8369e expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787826425.414971:0:55719:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9768b0163480 x1874670563992320/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 376/584 e 0 to 0 dl 1787826441 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787826425.414984:0:55719:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8369e expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787826425.415005:0:55719:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8369e expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.415021:0:55719:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:1.0:1787826425.415029:0:55719:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1787826425.415055:0:55719:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1728 00000080:00010000:1.0:1787826425.415066:0:55719:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1728 from it ffffb2d44225bbc8 00010000:00010000:1.0:1787826425.415071:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8369e expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.415084:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8369e expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826425.415117:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8369e expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826425.415129:0:55719:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8369e expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1787826425.415138:0:55719:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8369e expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826425.415202:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8369e expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826425.415211:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8369e expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787826425.415234:0:55719:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1787826425.415251:0:55719:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1787826425.415290:0:55719:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:1.0:1787826425.415304:0:55719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8369e expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826425.415314:0:55719:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8369e expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1787826425.415322:0:55719:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8369e expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787826425.415340:0:55719:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1787826425.415345:0:55719:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1787826425.415354:0:55719:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000008:00010000:1.0:1787826425.415370:0:55719:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 0 [cma]time: 1787826424 1787826412 1787826412 by oinfo size 0 blocks 0 [cma]time 1787826425 1787826412 1787826412 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 1/0,0 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b83689 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787826425.415449:0:55719:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 4294967294 -> 36 00010000:00010000:1.0:1787826425.415455:0:55719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b8369e expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826425.415462:0:55719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8369e expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:1.0:1787826425.415501:0:55719:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 1154002 00000002:00100000:1.0:1787826425.415565:0:55719:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889efbc00 x1874670563986816/t8589935217(8589935217) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826440 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1787826425.415578:0:55719:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1787826425.415585:0:55719:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b0163480 x1874670563992576/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1787826425.415612:0:55719:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0163480 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563992576:192.168.203.139@tcp:35:lfs.0 00000100:00100000:1.0:1787826425.415710:0:55719:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3600 going to sleep for 11 seconds 00000100:00100000:1.0:1787826425.431255:0:55719:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0163480 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563992576:192.168.203.139@tcp:35:lfs.0 00000002:00100000:1.0:1787826425.431304:0:55719:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889efbc00 x1874670563986816/t8589935217(8589935217) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826440 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1787826425.431408:0:55719:0:(file.c:479:ll_file_release()) START file . :VOLATILE:0000:7BD4BC4A:fd=03:[0x200000405:0x40:0x0](ffff976890b3db88), flags 554001 00000002:00100000:1.0:1787826425.431472:0:55719:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889ef9a40 x1874670563987200/t8589935218(8589935218) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826441 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1787826425.431483:0:55719:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1787826425.431489:0:55719:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b0163480 x1874670563992832/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1787826425.431514:0:55719:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0163480 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563992832:192.168.203.139@tcp:35:lfs.0 00000100:00100000:1.0:1787826425.431749:0:55719:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3600 going to sleep for 11 seconds 00000100:00100000:1.0:1787826425.438071:0:55719:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0163480 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55719:1874670563992832:192.168.203.139@tcp:35:lfs.0 00000002:00100000:1.0:1787826425.438118:0:55719:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889ef9a40 x1874670563987200/t8589935218(8589935218) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826441 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1787826425.456588:0:55721:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:2.0:1787826425.456634:0:55721:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826425.456757:0:55721:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826425.456777:0:55721:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826425.456787:0:55721:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826425.456797:0:55721:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:2.0:1787826425.456802:0:55721:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:2.0:1787826425.456812:0:55721:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4421d3b58 00010000:00010000:2.0:1787826425.456817:0:55721:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826425.456842:0:55721:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826425.456862:0:55721:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:2.0:1787826425.456872:0:55721:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=24 00000080:00200000:2.0:1787826425.456878:0:55721:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787826425.456892:0:55721:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), flags=8 00000080:00200000:2.0:1787826425.456897:0:55721:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=getattr 00000002:00010000:2.0:1787826425.456915:0:55721:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f60b.sanity-flr,[0x0:0x0:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787826425.456980:0:55721:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55721 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.456989:0:55721:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55721 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.457001:0:55721:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55721 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826425.457040:0:55721:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8000 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55721:1874670563992960:192.168.203.139@tcp:101:getfattr.0 00000100:00100000:2.0:1787826425.457165:0:55721:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35540 going to sleep for 11 seconds 00000100:00100000:3.0:1787826425.463804:0:55721:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8000 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55721:1874670563992960:192.168.203.139@tcp:101:getfattr.0 00010000:00010000:3.0:1787826425.463883:0:55721:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 4/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.463915:0:55721:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.463925:0:55721:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.463939:0:55721:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.463951:0:55721:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.463967:0:55721:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.463976:0:55721:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.463984:0:55721:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826425.463998:0:55721:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff976889ef8000 x1874670563992960/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 584/824 e 0 to 0 dl 1787826441 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'getfattr.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826425.464115:0:55721:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826425.464131:0:55721:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.464143:0:55721:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826425.464151:0:55721:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1787826425.464223:0:55721:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x1d:0x0], updating i_size 0 00000080:00200000:3.0:1787826425.464227:0:55721:0:(namei.c:147:ll_iget()) got inode: [0x200000405:0x1d:0x0](ffff9768901a80c8): rc = 0 00000080:00200000:3.0:1787826425.464244:0:55721:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 36 -> 36 00000080:00010000:3.0:1787826425.464251:0:55721:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab172f 00000080:00010000:3.0:1787826425.464264:0:55721:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:3.0:1787826425.464268:0:55721:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab172f from it ffffb2d4421d3b20 00010000:00010000:3.0:1787826425.464272:0:55721:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.464283:0:55721:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1787826425.464330:0:55721:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826425.464338:0:55721:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:3.0:1787826425.464479:0:55721:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826425.464490:0:55721:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.464502:0:55721:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826425.464509:0:55721:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826425.464514:0:55721:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:3.0:1787826425.464518:0:55721:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab172f 00000080:00010000:3.0:1787826425.464522:0:55721:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab172f from it ffffb2d4421d3cb8 00010000:00010000:3.0:1787826425.464525:0:55721:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.464532:0:55721:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826425.464584:0:55721:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826425.464593:0:55721:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826425.464601:0:55721:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826425.464608:0:55721:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826425.464614:0:55721:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826425.464618:0:55721:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826425.464621:0:55721:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826425.464624:0:55721:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4421d3bf8 00010000:00010000:3.0:1787826425.464627:0:55721:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826425.464635:0:55721:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826425.464643:0:55721:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826425.464651:0:55721:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:3.0:1787826425.464656:0:55721:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787826425.464659:0:55721:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=9 00000080:00200000:3.0:1787826425.464666:0:55721:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:3.0:1787826425.464673:0:55721:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200000:3.0:1787826425.464692:0:55721:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826425.464698:0:55721:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826425.464706:0:55721:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826425.464713:0:55721:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826425.464720:0:55721:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826425.464724:0:55721:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826425.464726:0:55721:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826425.464730:0:55721:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4421d3bf8 00010000:00010000:3.0:1787826425.464733:0:55721:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826425.464741:0:55721:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826425.464755:0:55721:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826425.464758:0:55721:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:3.0:1787826425.464761:0:55721:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787826425.464764:0:55721:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=9 00000080:00200000:3.0:1787826425.464774:0:55721:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:3.0:1787826425.464777:0:55721:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200000:2.0:1787826425.508941:0:55725:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:2.0:1787826425.508981:0:55725:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826425.509005:0:55725:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826425.509024:0:55725:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826425.509033:0:55725:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826425.509043:0:55725:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:2.0:1787826425.509047:0:55725:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:2.0:1787826425.509057:0:55725:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44222bbf8 00010000:00010000:2.0:1787826425.509062:0:55725:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826425.509073:0:55725:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826425.509091:0:55725:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:2.0:1787826425.509100:0:55725:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=27 00000080:00200000:2.0:1787826425.509106:0:55725:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787826425.509110:0:55725:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=267 00000080:00200000:2.0:1787826425.509144:0:55725:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:2.0:1787826425.509152:0:55725:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826425.509160:0:55725:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826425.509169:0:55725:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826425.509176:0:55725:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826425.509180:0:55725:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:2.0:1787826425.509196:0:55725:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:2.0:1787826425.509201:0:55725:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44222bbf8 00010000:00010000:2.0:1787826425.509204:0:55725:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826425.509212:0:55725:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826425.509221:0:55725:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:2.0:1787826425.509226:0:55725:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=267 00000080:00200000:2.0:1787826425.509229:0:55725:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 44 00000080:00200008:2.0:1787826425.509236:0:55725:0:(file.c:1090:ll_file_open()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00000080:00200080:2.0:1787826425.509270:0:55725:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8) cmd=c0086915 arg=10693d0 00000100:00100000:2.0:1787826425.509361:0:55725:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef92c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55725:1874670563993088:192.168.203.139@tcp:34:lfs.0 00000100:00100000:2.0:1787826425.509534:0:55725:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35300 going to sleep for 11 seconds 00000100:00100000:2.0:1787826425.520051:0:55725:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef92c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55725:1874670563993088:192.168.203.139@tcp:34:lfs.0 00000080:00200008:2.0:1787826425.520571:0:55725:0:(file.c:479:ll_file_release()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00010000:00010000:2.0:1787826425.520613:0:55725:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9600/0x8e7c99ee62ab0c1c lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b818c9 expref: -99 pid: 55483 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00200000:0.0:1787826425.549120:0:55727:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826425.549239:0:55727:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826425.549266:0:55727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826425.549287:0:55727:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826425.549296:0:55727:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826425.549306:0:55727:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826425.549309:0:55727:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826425.549318:0:55727:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44226bbf8 00010000:00010000:0.0:1787826425.549322:0:55727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826425.549331:0:55727:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826425.549349:0:55727:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826425.549359:0:55727:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:0.0:1787826425.549366:0:55727:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826425.549371:0:55727:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:0.0:1787826425.549374:0:55727:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 44 00000080:00200008:0.0:1787826425.549380:0:55727:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154400 00000002:00010000:0.0:1787826425.549401:0:55727:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114401 00010000:00010000:0.0:1787826425.549465:0:55727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1736 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 55727 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.549473:0:55727:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1736 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55727 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826425.549493:0:55727:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826425.549508:0:55727:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aeed00 x1874670563993344/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826425.549518:0:55727:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1736 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55727 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826425.549545:0:55727:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563993344:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826425.550795:0:55727:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e540 going to sleep for 11 seconds 00000100:00100000:2.0:1787826425.554391:0:55727:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563993344:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826425.554431:0:55727:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1736 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b836e4 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.554449:0:55727:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1736 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b836e4 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.554462:0:55727:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1736 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b836e4 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826425.554486:0:55727:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff976882aeed00 x1874670563993344/t8589935230(8589935230) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787826425.554501:0:55727:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aeed00 x1874670563993344/t8589935230(8589935230) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826425.554516:0:55727:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1736 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b836e4 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787826425.554530:0:55727:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1736 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b836e4 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.554545:0:55727:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:2.0:1787826425.554551:0:55727:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787826425.554607:0:55727:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x1d:0x0], updating i_size 0 00000080:00200000:2.0:1787826425.554622:0:55727:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 36 -> 36 00000080:00010000:2.0:1787826425.554627:0:55727:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1736 00000080:00010000:2.0:1787826425.554635:0:55727:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1736 from it ffffb2d44226bbf0 00010000:00010000:2.0:1787826425.554639:0:55727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1736 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b836e4 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.554650:0:55727:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1736 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b836e4 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000002:00100000:2.0:1787826425.554684:0:55727:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aeed00 x1874670563993344/t8589935230(8589935230) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1787826425.554975:0:55727:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:2.0:1787826425.554986:0:55727:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:2.0:1787826425.555019:0:55727:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787826425.555027:0:55727:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:2.0:1787826425.555053:0:55727:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826425.555065:0:55727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826425.555078:0:55727:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826425.555086:0:55727:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826425.555092:0:55727:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:2.0:1787826425.555096:0:55727:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab172f 00000080:00010000:2.0:1787826425.555100:0:55727:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab172f from it ffffb2d44226bcd8 00010000:00010000:2.0:1787826425.555103:0:55727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826425.555112:0:55727:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:2.0:1787826425.555161:0:55727:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154000 00010000:00010000:2.0:1787826425.555173:0:55727:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1736 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b836e4 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826425.555308:0:55727:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:2.0:1787826425.555317:0:55727:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826425.555326:0:55727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826425.555341:0:55727:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826425.555347:0:55727:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826425.555351:0:55727:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:2.0:1787826425.555355:0:55727:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:2.0:1787826425.555359:0:55727:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44226bbf8 00010000:00010000:2.0:1787826425.555362:0:55727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826425.555370:0:55727:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826425.555378:0:55727:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:2.0:1787826425.555385:0:55727:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:2.0:1787826425.555389:0:55727:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787826425.555393:0:55727:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:2.0:1787826425.555397:0:55727:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 44 00000080:00200008:2.0:1787826425.555405:0:55727:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154400 00000080:00200000:2.0:1787826425.555434:0:55727:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:2.0:1787826425.555438:0:55727:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:2.0:1787826425.555452:0:55727:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787826425.555455:0:55727:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:2.0:1787826425.555463:0:55727:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826425.555470:0:55727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826425.555483:0:55727:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787826425.555489:0:55727:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826425.555494:0:55727:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:2.0:1787826425.555497:0:55727:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab172f 00000080:00010000:2.0:1787826425.555501:0:55727:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab172f from it ffffb2d44226bcd8 00010000:00010000:2.0:1787826425.555504:0:55727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826425.555511:0:55727:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200008:2.0:1787826425.555530:0:55727:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154000 00010000:00010000:2.0:1787826425.555539:0:55727:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1736 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b836e4 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787826425.555572:0:55727:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:2.0:1787826425.555579:0:55727:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826425.555586:0:55727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826425.555594:0:55727:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826425.555601:0:55727:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826425.555605:0:55727:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:2.0:1787826425.555608:0:55727:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:2.0:1787826425.555611:0:55727:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44226bbf8 00010000:00010000:2.0:1787826425.555615:0:55727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826425.555621:0:55727:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826425.555629:0:55727:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:2.0:1787826425.555633:0:55727:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:2.0:1787826425.555636:0:55727:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787826425.555639:0:55727:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:2.0:1787826425.555641:0:55727:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 46 00000080:00200008:2.0:1787826425.555646:0:55727:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154402 00000002:00010000:2.0:1787826425.555656:0:55727:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114403 00010000:00010000:2.0:1787826425.555718:0:55727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab173d lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 55727 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.555725:0:55727:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab173d lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55727 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1787826425.555738:0:55727:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1787826425.555748:0:55727:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889ef92c0 x1874670563993600/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1787826425.555758:0:55727:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab173d lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55727 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826425.555782:0:55727:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef92c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563993600:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826425.556655:0:55727:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35300 going to sleep for 11 seconds 00000100:00100000:0.0:1787826425.559991:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826425.560099:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585289344 00000100:00100000:1.0:1787826425.560111:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585289344 00000100:00100000:1.0:1787826425.560219:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 589 00000100:00100000:1.0:1787826425.560232:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aeda40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:32404:x1874670585289344:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826425.560256:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787826425.560356:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00010000:00010000:3.0:1787826425.560437:0:49796:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826425.560447:0:49796:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826425.560456:0:49796:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515400 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826425.560461:0:49796:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826425.560471:0:49796:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826425.560480:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826425.560513:0:49796:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1736 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b836e4 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826425.560565:0:49796:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 2/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826425.560572:0:49796:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826425.560605:0:49796:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 1/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826425.560612:0:49796:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515400/0x8e7c99ee62ab172f lrc: 0/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b836c1 expref: -99 pid: 55721 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826425.560630:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826425.560632:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826425.560634:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826425.560644:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826425.560661:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563993728:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826425.560948:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aeda40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:32404:x1874670585289344:12345-192.168.203.139@tcp:104: Request processed in 718us (972us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826425.560958:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 589 00000100:00100000:3.0:1787826425.567387:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563993728:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826425.567643:0:55727:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef92c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563993600:192.168.203.139@tcp:101:lfs.0 00010000:00010000:1.0:1787826425.567671:0:55727:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode CW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab173d lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b836f9 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.567686:0:55727:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab173d lrc: 4/1,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b836f9 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.567701:0:55727:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab173d lrc: 4/1,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b836f9 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.567714:0:55727:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab173d lrc: 4/1,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b836f9 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.567731:0:55727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab173d lrc: 5/1,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b836f9 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.567739:0:55727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab173d lrc: 5/1,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b836f9 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.567746:0:55727:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab173d lrc: 4/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b836f9 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787826425.567758:0:55727:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff976889ef92c0 x1874670563993600/t8589935231(8589935231) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1787826425.567772:0:55727:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889ef92c0 x1874670563993600/t8589935231(8589935231) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787826425.567789:0:55727:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab173d lrc: 3/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b836f9 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.567806:0:55727:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 8 res 8589935621/29 (0 0) 00000080:00200000:1.0:1787826425.567814:0:55727:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1787826425.567909:0:55727:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x1d:0x0], updating i_size 0 00000080:00010000:1.0:1787826425.567916:0:55727:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab173d 00000080:00010000:1.0:1787826425.567923:0:55727:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab173d from it ffffb2d44226bbf0 00010000:00010000:1.0:1787826425.567927:0:55727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab173d lrc: 3/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b836f9 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.567937:0:55727:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab173d lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b836f9 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:1.0:1787826425.567973:0:55727:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889ef92c0 x1874670563993600/t8589935231(8589935231) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:1.0:1787826425.568124:0:55727:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=800466af arg=7fffb1f75bdc 00000080:00200080:1.0:1787826425.568144:0:55727:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=80086601 arg=7fffb1f75b78 00000100:00100000:1.0:1787826425.568207:0:55727:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0161a40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563993856:192.168.203.139@tcp:33:lfs.0 00000100:00100000:1.0:1787826425.568306:0:55727:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3900 going to sleep for 11 seconds 00000100:00100000:3.0:1787826425.570921:0:55727:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0161a40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563993856:192.168.203.139@tcp:33:lfs.0 00000080:00200080:3.0:1787826425.570978:0:55727:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=7fffb1f75b64 00000002:00010000:3.0:1787826425.571013:0:55727:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 01004040000002 00010000:00010000:3.0:1787826425.571038:0:55727:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab173d lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x22153cd396b836f9 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.571079:0:55727:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1736 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b836e4 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826425.571094:0:55727:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8369e expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000002:00100000:3.0:1787826425.571144:0:55727:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976882aeed00 x1874670563993344/t8589935230(8589935230) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826441 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826425.571156:0:55727:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1787826425.571163:0:55727:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff97688cb5f840 x1874670563994112/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826425.571196:0:55727:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563994112:192.168.203.139@tcp:35:lfs.0 00000100:00100000:3.0:1787826425.571274:0:55727:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25900 going to sleep for 11 seconds 00000080:00200000:2.0:1787826425.573236:0:42426:0:(vvp_io.c:1889:vvp_io_init()) [0x240000404:0x11:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787826425.573258:0:42426:0:(lov_io.c:693:lov_io_mirror_init()) [0x240000404:0x11:0x0]: flr state: 1, move mirror from 1 to 1, have retried: 0, mirror count: 2 00020000:00200000:2.0:1787826425.573262:0:42426:0:(lov_io.c:750:lov_io_mirror_init()) use delayed RPC state for this IO 00020000:00200000:2.0:1787826425.573274:0:42426:0:(lov_io.c:1253:lov_io_iter_init()) component[2] flags 0x10 00020000:00200000:2.0:1787826425.573289:0:42426:0:(lov_io.c:1253:lov_io_iter_init()) component[3] flags 0x10 00000080:00200000:2.0:1787826425.573393:0:42426:0:(vvp_io.c:304:vvp_io_fini()) [0x240000404:0x11:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1787826425.578865:0:55727:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563994112:192.168.203.139@tcp:35:lfs.0 00000002:00100000:3.0:1787826425.578914:0:55727:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976882aeed00 x1874670563993344/t8589935230(8589935230) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826441 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1787826425.579038:0:55727:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab173d lrc: 1/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b836f9 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.579048:0:55727:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1736 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b836e4 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826425.579055:0:55727:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1787826425.579058:0:55727:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab173d lrc: 0/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b836f9 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.579072:0:55727:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab1736 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b836e4 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826425.579100:0:55727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab1744 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 55727 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.579108:0:55727:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x81000000001000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab1744 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 55727 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787826425.579120:0:55727:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787826425.579128:0:55727:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff97688cb5f840 x1874670563994368/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 976/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787826425.579138:0:55727:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab1744 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x81000000000000 pid: 55727 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826425.579162:0:55727:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563994368:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826425.580204:0:55727:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25900 going to sleep for 11 seconds 00000100:00100000:2.0:1787826425.592233:0:55727:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563994368:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826425.592271:0:55727:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab1744 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b8370e expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.592286:0:55727:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab1744 lrc: 4/1,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b8370e expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.592300:0:55727:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab1744 lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b8370e expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.592313:0:55727:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab1744 lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b8370e expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.592327:0:55727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab1744 lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b8370e expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.592337:0:55727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab1744 lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b8370e expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.592345:0:55727:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab1744 lrc: 4/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b8370e expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826425.592358:0:55727:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x600002b, status=0 req@ffff97688cb5f840 x1874670563994368/t8589935233(8589935233) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 976/848 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787826425.592372:0:55727:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff97688cb5f840 x1874670563994368/t8589935233(8589935233) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 976/848 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826425.592389:0:55727:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab1744 lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b8370e expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.592405:0:55727:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 2 res 8589935621/29 (0 0) 00000080:00200000:2.0:1787826425.592412:0:55727:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000002:00100000:2.0:1787826425.592439:0:55727:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff97688cb5f840 x1874670563994368/t8589935233(8589935233) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 768/848 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1787826425.592448:0:55727:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1744 00000080:00010000:2.0:1787826425.592455:0:55727:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1744 from it ffffb2d44226bcc0 00010000:00010000:2.0:1787826425.592459:0:55727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab1744 lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b8370e expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.592466:0:55727:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab1744 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b8370e expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200080:2.0:1787826425.592557:0:55727:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=1f742e0 00010000:00010000:2.0:1787826425.592626:0:55727:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab1744 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0x22153cd396b8370e expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.592643:0:55727:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9800/0x8e7c99ee62ab1744 lrc: 0/0,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x85809400000000 nid: local remote: 0x22153cd396b8370e expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826425.592669:0:55727:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff97688cb5f840 x1874670563994368/t8589935233(8589935233) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 768/848 e 0 to 0 dl 1787826441 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787826425.592680:0:55727:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787826425.592687:0:55727:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889ef8f00 x1874670563994624/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 504/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787826425.592713:0:55727:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8f00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563994624:192.168.203.139@tcp:35:lfs.0 00000100:00100000:2.0:1787826425.592806:0:55727:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826425.598648:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826425.599062:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585289472 00000100:00100000:1.0:1787826425.599076:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585289472 00000100:00100000:1.0:1787826425.599155:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 590 00000100:00100000:1.0:1787826425.599167:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aee1c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6522:x1874670585289472:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826425.599199:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b8369e expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787826425.599304:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826425.599310:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aee1c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6522:x1874670585289472:12345-192.168.203.139@tcp:104: Request processed in 147us (688us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826425.599319:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 590 00010000:00010000:2.0:1787826425.599568:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826425.599578:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b8369e expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826425.599653:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976890fdee00 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:2.0:1787826425.599659:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b8369e expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826425.599669:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b8369e expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826425.599678:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b8369e expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1787826425.599706:0:42220:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x1d:0x0]: losing layout lock 00010000:00010000:2.0:1787826425.599758:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b8369e expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826425.599766:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826425.599796:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b8369e expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826425.599804:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab1728 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b8369e expref: -99 pid: 55719 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000100:00100000:3.0:1787826425.599867:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563994752:192.168.203.139@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1787826425.599902:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826425.599905:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826425.599907:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826425.608553:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563994752:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826425.609929:0:55727:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8f00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563994624:192.168.203.139@tcp:35:lfs.0 00000080:00020000:2.0:1787826425.609950:0:55727:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9768877a4800: inode [0x200000405:0x1d:0x0] mdc close failed: rc = -22 00000002:00100000:2.0:1787826425.609974:0:55727:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff97688cb5f840 x1874670563994368/t8589935233(8589935233) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 768/848 e 0 to 0 dl 1787826441 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1787826425.610017:0:55727:0:(file.c:7076:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000002:00010000:2.0:1787826425.610032:0:55727:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: layout flags 0 00010000:00010000:2.0:1787826425.610097:0:55727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab174b lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55727 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.610107:0:55727:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab174b lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55727 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.610117:0:55727:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab174b lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55727 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826425.610139:0:55727:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa580 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563994880:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826425.610246:0:55727:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35c00 going to sleep for 11 seconds 00000100:00100000:2.0:1787826425.613638:0:55727:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa580 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563994880:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826425.613679:0:55727:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab174b lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83723 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.613696:0:55727:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab174b lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83723 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826425.613712:0:55727:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff976889efa580 x1874670563994880/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 376/584 e 0 to 0 dl 1787826441 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826425.613724:0:55727:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab174b lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83723 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787826425.613738:0:55727:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab174b lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83723 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.613751:0:55727:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:2.0:1787826425.613759:0:55727:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1787826425.613779:0:55727:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab174b 00000080:00010000:2.0:1787826425.613789:0:55727:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab174b from it ffffb2d44226bbc8 00010000:00010000:2.0:1787826425.613793:0:55727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab174b lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83723 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.613804:0:55727:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab174b lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83723 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.613861:0:55727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab174b lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83723 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.613874:0:55727:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab174b lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83723 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1787826425.613888:0:55727:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab174b lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83723 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826425.613941:0:55727:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 4294967294 -> 36 00010000:00010000:2.0:1787826425.613948:0:55727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab174b lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83723 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.613956:0:55727:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab174b lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83723 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826425.614215:0:55727:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:2.0:1787826425.614240:0:55727:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826425.614251:0:55727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826425.614261:0:55727:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826425.614268:0:55727:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826425.614274:0:55727:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:2.0:1787826425.614278:0:55727:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:2.0:1787826425.614282:0:55727:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44226bbf8 00010000:00010000:2.0:1787826425.614285:0:55727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826425.614292:0:55727:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826425.614302:0:55727:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:2.0:1787826425.614311:0:55727:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=24 00000080:00200000:2.0:1787826425.614317:0:55727:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787826425.614335:0:55727:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 3 00000080:00200000:2.0:1787826425.614339:0:55727:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=. :VOLATILE:0000:26E7C289:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), file ffff9768850b3800, open_flags 2f9c2, mode 8180 opened 0 00000080:00200000:2.0:1787826425.614346:0:55727:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=. :VOLATILE:0000:26E7C289:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=open|creat 00000002:00010000:2.0:1787826425.614371:0:55727:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: . :VOLATILE:0000:26E7C289:fd=03,[0x200000405:0x41:0x0]) in obj [0x200000bd0:0x4:0x0], intent: open|creat flags 0574703 00010000:00010000:2.0:1787826425.614429:0:55727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1752 lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55727 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.614437:0:55727:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1752 lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55727 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1787826425.614449:0:55727:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1787826425.614457:0:55727:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efa580 x1874670563995136/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/66640 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1787826425.614467:0:55727:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1752 lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55727 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826425.614491:0:55727:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa580 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563995136:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826425.614652:0:55727:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35c00 going to sleep for 11 seconds 00000100:00100000:2.0:1787826425.619791:0:55727:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa580 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563995136:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826425.619858:0:55727:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1752 lrc: 4/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83731 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.619902:0:55727:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1752 lrc: 4/0,1 mode: --/PR res: [0x200000405:0x41:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83731 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.619912:0:55727:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1752 lrc: 4/0,1 mode: --/PR res: [0x200000405:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83731 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.619925:0:55727:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1752 lrc: 4/0,1 mode: PR/PR res: [0x200000405:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83731 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.619938:0:55727:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1752 lrc: 4/0,1 mode: PR/PR res: [0x200000405:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83731 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.619954:0:55727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1752 lrc: 5/1,1 mode: PR/PR res: [0x200000405:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83731 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.619963:0:55727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1752 lrc: 5/1,1 mode: PR/PR res: [0x200000405:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83731 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.619971:0:55727:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1752 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83731 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826425.619985:0:55727:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff976889efa580 x1874670563995136/t8589935235(8589935235) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787826425.619998:0:55727:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efa580 x1874670563995136/t8589935235(8589935235) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826425.620011:0:55727:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1752 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83731 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.620024:0:55727:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/65 (0 0) 00000080:00200000:2.0:1787826425.620032:0:55727:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd0:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787826425.620048:0:55727:0:(namei.c:1628:ll_create_it()) VFS Op:name=. :VOLATILE:0000:26E7C289:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=open|creat 00000080:00200000:2.0:1787826425.620083:0:55727:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000405:0x41:0x0](ffff976890b3cb08) 00000080:00200000:2.0:1787826425.620091:0:55727:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x41:0x0], updating i_size 0 00000080:00200000:2.0:1787826425.620130:0:55727:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x41:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1787826425.620135:0:55727:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000405:0x41:0x0](ffff976890b3cb08) 00000080:00010000:2.0:1787826425.620138:0:55727:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x41:0x0](ffff976890b3cb08) for lock 0x8e7c99ee62ab1752 00000080:00010000:2.0:1787826425.620310:0:55727:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x41:0x0](ffff976890b3cb08) 00000080:00200008:2.0:1787826425.620329:0:55727:0:(file.c:1090:ll_file_open()) START file . :VOLATILE:0000:26E7C289:fd=03:[0x200000405:0x41:0x0](ffff976890b3cb08), flags 574702 00000002:00100000:2.0:1787826425.620353:0:55727:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efa580 x1874670563995136/t8589935235(8589935235) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1787826425.620396:0:55727:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1752 from it ffff9768876c9360 00010000:00010000:2.0:1787826425.620402:0:55727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1752 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83731 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.620413:0:55727:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1752 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83731 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826425.620440:0:55727:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x41:0x0](ffff976890b3cb08), inode mode 8180 mask 40 00000080:00200080:2.0:1787826425.620470:0:55727:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=7fffb1f75b64 00000002:00010000:2.0:1787826425.620494:0:55727:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 01004040000002 00010000:00010000:2.0:1787826425.621306:0:55727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab1759 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 55727 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.621319:0:55727:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x81000000001000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab1759 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 55727 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1787826425.621334:0:55727:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1787826425.621343:0:55727:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889ef8000 x1874670563995392/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1787826425.621353:0:55727:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab1759 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x81000000000000 pid: 55727 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826425.621379:0:55727:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8000 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563995392:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826425.622105:0:55727:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35c00 going to sleep for 11 seconds 00000100:00100000:2.0:1787826425.634257:0:55727:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8000 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563995392:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826425.634293:0:55727:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab1759 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83746 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.634308:0:55727:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab1759 lrc: 4/1,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83746 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.634322:0:55727:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab1759 lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83746 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.634333:0:55727:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab1759 lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83746 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.634350:0:55727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab1759 lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83746 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.634702:0:55727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab1759 lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83746 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.634716:0:55727:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab1759 lrc: 4/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83746 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826425.634731:0:55727:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x600002b, status=0 req@ffff976889ef8000 x1874670563995392/t8589935236(8589935236) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787826425.634745:0:55727:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889ef8000 x1874670563995392/t8589935236(8589935236) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826425.634762:0:55727:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab1759 lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83746 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.634777:0:55727:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 2 res 8589935621/29 (0 0) 00000080:00200000:2.0:1787826425.634784:0:55727:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000002:00100000:2.0:1787826425.634816:0:55727:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889ef8000 x1874670563995392/t8589935236(8589935236) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1787826425.634851:0:55727:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1759 00000080:00010000:2.0:1787826425.634859:0:55727:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1759 from it ffffb2d44226bcc0 00010000:00010000:2.0:1787826425.634862:0:55727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab1759 lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83746 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.634870:0:55727:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab1759 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83746 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200080:2.0:1787826425.635477:0:55727:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=1f74300 00010000:00010000:2.0:1787826425.635547:0:55727:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab1759 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0x22153cd396b83746 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.635567:0:55727:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab1759 lrc: 0/0,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x85809400000000 nid: local remote: 0x22153cd396b83746 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826425.635589:0:55727:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889ef8000 x1874670563995392/t8589935236(8589935236) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826441 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787826425.635601:0:55727:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787826425.635608:0:55727:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889ef9a40 x1874670563995648/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 504/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787826425.636192:0:55727:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563995648:192.168.203.139@tcp:35:lfs.0 00000100:00100000:2.0:1787826425.636310:0:55727:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826425.639325:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826425.639542:0:42210:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1787826425.639566:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585289600 00000100:00100000:1.0:1787826425.639577:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585289600 00000100:00100000:1.0:1787826425.639710:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 591 00000100:00100000:1.0:1787826425.639723:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff97688a33e450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585289600:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826425.639745:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab174b lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83723 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826425.640143:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826425.640151:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff97688a33e450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585289600:12345-192.168.203.139@tcp:104: Request processed in 431us (838us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826425.640160:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 591 00010000:00010000:2.0:1787826425.641261:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826425.641278:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab174b lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83723 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826425.641290:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514200 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:2.0:1787826425.641296:0:42221:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab174b lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83723 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826425.641306:0:42221:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab174b lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83723 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826425.641316:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab174b lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b83723 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826425.641346:0:42221:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x1d:0x0]: losing layout lock 00010000:00010000:2.0:1787826425.641420:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab174b lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83723 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826425.641428:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826425.641457:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab174b lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83723 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826425.641465:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab174b lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83723 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826425.641494:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826425.641497:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826425.641498:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826425.641522:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563995776:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826425.646312:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826425.646420:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585289728 00000100:00100000:1.0:1787826425.646432:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585289728 00000100:00100000:1.0:1787826425.646494:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 592 00000100:00100000:1.0:1787826425.646505:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aef840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585289728:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826425.646526:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1752 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83731 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826425.646619:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826425.646632:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aef840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585289728:12345-192.168.203.139@tcp:104: Request processed in 124us (349us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826425.646641:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 592 00010000:00010000:3.0:1787826425.647081:0:49796:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826425.647090:0:49796:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1752 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83731 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.647100:0:49796:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514c00 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826425.647106:0:49796:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1752 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x41:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83731 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.647116:0:49796:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1752 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x41:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83731 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.647125:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1752 lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x41:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b83731 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826425.647154:0:49796:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x41:0x0]: losing layout lock 00010000:00010000:3.0:1787826425.647216:0:49796:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1752 lrc: 2/0,0 mode: --/PR res: [0x200000405:0x41:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83731 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.647223:0:49796:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826425.647244:0:49796:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1752 lrc: 1/0,0 mode: --/PR res: [0x200000405:0x41:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83731 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.647253:0:49796:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1752 lrc: 0/0,0 mode: --/PR res: [0x200000405:0x41:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83731 expref: -99 pid: 55727 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.647273:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826425.647274:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826425.647276:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826425.647294:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563995904:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826425.647688:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563995776:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826425.652219:0:55727:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563995648:192.168.203.139@tcp:35:lfs.0 00000002:00100000:2.0:1787826425.652270:0:55727:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889ef8000 x1874670563995392/t8589935236(8589935236) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826441 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1787826425.652322:0:55727:0:(file.c:7076:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000002:00010000:2.0:1787826425.652335:0:55727:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: layout flags 0 00010000:00010000:2.0:1787826425.652404:0:55727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55727 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.652414:0:55727:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55727 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.652423:0:55727:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55727 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826425.652445:0:55727:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563996032:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826425.652538:0:55727:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35c00 going to sleep for 11 seconds 00000100:00100000:3.0:1787826425.654322:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563995904:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826425.663424:0:55727:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563996032:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826425.663470:0:55727:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83762 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.663488:0:55727:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83762 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826425.663506:0:55727:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff976889ef9a40 x1874670563996032/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 376/480 e 0 to 0 dl 1787826441 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826425.663517:0:55727:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83762 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787826425.663531:0:55727:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83762 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.663599:0:55727:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:2.0:1787826425.663607:0:55727:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1787826425.663697:0:55727:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1760 00000080:00010000:2.0:1787826425.663709:0:55727:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1760 from it ffffb2d44226bbc8 00010000:00010000:2.0:1787826425.663714:0:55727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83762 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.663726:0:55727:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83762 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.663762:0:55727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83762 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.663775:0:55727:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83762 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1787826425.663784:0:55727:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83762 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.663853:0:55727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83762 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.663861:0:55727:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83762 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826425.663881:0:55727:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787826425.663894:0:55727:0:(lov_io.c:693:lov_io_mirror_init()) [0x200000405:0x1d:0x0]: flr state: 1, move mirror from 1 to 1, have retried: 0, mirror count: 2 00020000:00200000:2.0:1787826425.663897:0:55727:0:(lov_io.c:750:lov_io_mirror_init()) use delayed RPC state for this IO 00020000:00200000:2.0:1787826425.663975:0:55727:0:(lov_io.c:1253:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:2.0:1787826425.664075:0:55727:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:2.0:1787826425.664091:0:55727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83762 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826425.664100:0:55727:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83762 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1787826425.664107:0:55727:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83762 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826425.664124:0:55727:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787826425.664130:0:55727:0:(lov_io.c:693:lov_io_mirror_init()) [0x200000405:0x1d:0x0]: flr state: 1, move mirror from 1 to 1, have retried: 0, mirror count: 2 00020000:00200000:2.0:1787826425.664132:0:55727:0:(lov_io.c:750:lov_io_mirror_init()) use delayed RPC state for this IO 00020000:00200000:2.0:1787826425.664135:0:55727:0:(lov_io.c:1253:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:2.0:1787826425.664295:0:55727:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787826425.664444:0:55727:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 4294967294 -> 37 00010000:00010000:2.0:1787826425.664449:0:55727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83762 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826425.664457:0:55727:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83762 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:2.0:1787826425.664498:0:55727:0:(file.c:479:ll_file_release()) START file . :VOLATILE:0000:26E7C289:fd=03:[0x200000405:0x41:0x0](ffff976890b3cb08), flags 554002 00000002:00100000:2.0:1787826425.664608:0:55727:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889efa580 x1874670563995136/t8589935235(8589935235) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826441 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787826425.664621:0:55727:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787826425.664721:0:55727:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889ef9a40 x1874670563996288/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787826425.664752:0:55727:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563996288:192.168.203.139@tcp:35:lfs.0 00000100:00100000:2.0:1787826425.665139:0:55727:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35900 going to sleep for 11 seconds 00000100:00100000:2.0:1787826425.669166:0:55727:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563996288:192.168.203.139@tcp:35:lfs.0 00000002:00100000:2.0:1787826425.669230:0:55727:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889efa580 x1874670563995136/t8589935235(8589935235) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826441 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1787826425.669327:0:55727:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154002 00000002:00100000:2.0:1787826425.669407:0:55727:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889ef92c0 x1874670563993600/t8589935231(8589935231) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826441 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787826425.669418:0:55727:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787826425.669424:0:55727:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889ef8000 x1874670563996544/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787826425.669450:0:55727:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8000 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563996544:192.168.203.139@tcp:35:lfs.0 00000100:00100000:2.0:1787826425.669723:0:55727:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35900 going to sleep for 11 seconds 00000100:00100000:2.0:1787826425.677802:0:55727:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8000 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55727:1874670563996544:192.168.203.139@tcp:35:lfs.0 00000002:00100000:2.0:1787826425.677987:0:55727:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889ef92c0 x1874670563993600/t8589935231(8589935231) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826441 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1787826425.704337:0:55729:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826425.705148:0:55729:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826425.705174:0:55729:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826425.705203:0:55729:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826425.705213:0:55729:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826425.705225:0:55729:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826425.705229:0:55729:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826425.705240:0:55729:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44224bb58 00010000:00010000:1.0:1787826425.705245:0:55729:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826425.705255:0:55729:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826425.705275:0:55729:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826425.705285:0:55729:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=24 00000080:00200000:1.0:1787826425.705292:0:55729:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826425.705306:0:55729:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), flags=8 00000080:00200000:1.0:1787826425.705311:0:55729:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=getattr 00000002:00010000:1.0:1787826425.705328:0:55729:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f60b.sanity-flr,[0x0:0x0:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787826425.705968:0:55729:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55729 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.705979:0:55729:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55729 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826425.705992:0:55729:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55729 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826425.706454:0:55729:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0163c00 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55729:1874670563996672:192.168.203.139@tcp:101:getfattr.0 00000100:00100000:1.0:1787826425.707996:0:55729:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3600 going to sleep for 11 seconds 00000100:00100000:2.0:1787826425.720608:0:55729:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0163c00 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55729:1874670563996672:192.168.203.139@tcp:101:getfattr.0 00010000:00010000:2.0:1787826425.720758:0:55729:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 4/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.720792:0:55729:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.720802:0:55729:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.720816:0:55729:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.721098:0:55729:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.721119:0:55729:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.721128:0:55729:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.721137:0:55729:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826425.721154:0:55729:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9768b0163c00 x1874670563996672/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 584/720 e 0 to 0 dl 1787826441 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'getfattr.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826425.721169:0:55729:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826425.721560:0:55729:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.721577:0:55729:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/29 (0 0) 00000080:00200000:0.0:1787826425.721713:0:55729:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826425.721776:0:55729:0:(namei.c:147:ll_iget()) got inode: [0x200000405:0x1d:0x0](ffff9768901a80c8): rc = 0 00000080:00200000:0.0:1787826425.721793:0:55729:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 37 -> 37 00000080:00010000:0.0:1787826425.721800:0:55729:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1767 00000080:00010000:0.0:1787826425.721813:0:55729:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:0.0:1787826425.721817:0:55729:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1767 from it ffffb2d44224bb20 00010000:00010000:0.0:1787826425.721821:0:55729:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.725246:0:55729:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1787826425.725315:0:55729:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826425.725324:0:55729:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:0.0:1787826425.725467:0:55729:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826425.725551:0:55729:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826425.725566:0:55729:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826425.725575:0:55729:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826425.726252:0:55729:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:0.0:1787826425.726260:0:55729:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1767 00000080:00010000:0.0:1787826425.726271:0:55729:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1767 from it ffffb2d44224bcb8 00010000:00010000:0.0:1787826425.726276:0:55729:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826425.726288:0:55729:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826425.726320:0:55729:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 35 restore needed 0 00000001:00010000:0.0:1787826425.726333:0:55729:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:0.0:1787826425.726449:0:55729:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826425.726454:0:55729:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e55230, osc lock ffff9768849a3110, flags 0x400001002 00010000:00010000:0.0:1787826425.726542:0:55729:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826425.726566:0:55729:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826425.726576:0:55729:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826425.726687:0:55729:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826425.726741:0:55729:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab176e lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55729 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.726750:0:55729:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab176e lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55729 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826425.726977:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563996800:192.168.203.139@tcp:101:getfattr.0 00000020:00200000:0.0:1787826425.727100:0:55729:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1787826425.731069:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826425.733801:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585289984 00000100:00100000:0.0:1787826425.734017:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585289984 00000100:00100000:0.0:1787826425.734146:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 593 00000100:00100000:0.0:1787826425.734158:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0160f00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:8545:x1874670585289984:12345-192.168.203.139@tcp:106: 00010000:00010000:0.0:1787826425.734188:0:42210:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826425.734203:0:42210:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:0.0:1787826425.734242:0:42210:0:(cl_object.c:322:cl_object_glimpse()) header@000000003787b4ec[0x110, 1, [0x200000405:0x40:0x0] hash] 00000020:00010000:0.0:1787826425.734245:0:42210:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826425 blocks: 0 00000020:00010000:0.0:1787826425.734248:0:42210:0:(cl_object.c:322:cl_object_glimpse()) header@000000008b99b3d0[0x116, 2, [0x2c0000401:0x10e:0x0]] 00000020:00010000:0.0:1787826425.734250:0:42210:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826425 blocks: 0 00000100:00100000:0.0:1787826425.734344:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0160f00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:8545:x1874670585289984:12345-192.168.203.139@tcp:106: Request processed in 188us (4533us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826425.734354:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 593 00010000:00010000:0.0:1787826425.738544:0:2402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab176e lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55729 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.738576:0:2402:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab176e lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55729 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.738704:0:2402:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab176e lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55729 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.738714:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab176e lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55729 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.738723:0:2402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab176e lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55729 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826425.738740:0:2402:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:0.0:1787826425.738749:0:2402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1787826425.738767:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab176e lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55729 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.738777:0:2402:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab176e lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55729 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.738787:0:2402:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab176e lrc: 4/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55729 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.738796:0:2402:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925ca00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:0.0:1787826425.738803:0:2402:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab176e lrc: 5/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55729 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.738818:0:2402:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000080:00200000:1.0:1787826425.739007:0:55729:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:1.0:1787826425.739014:0:55729:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf0000 (ffff976890d10d88/0) at cl_lock_release():245 00000020:00010000:1.0:1787826425.739019:0:55729:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf0000 (ffff976890d10d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1787826425.739025:0:55729:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e55230 (ffff976890d10ea8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1787826425.739030:0:55729:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf0000 (ffff976890d10d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1787826425.739033:0:55729:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e55230 (ffff976890d10d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1787826425.739084:0:55729:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 37 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1787826425.739155:0:55729:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826425.739193:0:55729:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826425.739214:0:55729:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826425.739228:0:55729:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826425.739237:0:55729:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826425.739245:0:55729:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826425.739250:0:55729:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826425.739256:0:55729:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44224bbf8 00010000:00010000:1.0:1787826425.739261:0:55729:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826425.739270:0:55729:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826425.739289:0:55729:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826425.739297:0:55729:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:1.0:1787826425.739304:0:55729:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826425.739308:0:55729:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=9 00000080:00200000:1.0:1787826425.739319:0:55729:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:1.0:1787826425.739327:0:55729:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200000:1.0:1787826425.739351:0:55729:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826425.739359:0:55729:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826425.739366:0:55729:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826425.739375:0:55729:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826425.739381:0:55729:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826425.739385:0:55729:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826425.739388:0:55729:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826425.739392:0:55729:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44224bbf8 00010000:00010000:1.0:1787826425.739395:0:55729:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826425.739402:0:55729:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826425.739411:0:55729:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826425.739415:0:55729:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:1.0:1787826425.739418:0:55729:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826425.739421:0:55729:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=9 00000080:00200000:1.0:1787826425.739429:0:55729:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:1.0:1787826425.739432:0:55729:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00010000:00010000:0.0:1787826425.739575:0:2402:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab176e lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55729 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.739592:0:2402:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003f5af79d/0x8e7c99ee62ab176e lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55729 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826425.739618:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563996800:192.168.203.139@tcp:101:getfattr.0 00000080:00200000:1.0:1787826425.773198:0:55733:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826425.773242:0:55733:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826425.773267:0:55733:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826425.773425:0:55733:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826425.773437:0:55733:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826425.773446:0:55733:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826425.773451:0:55733:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826425.773459:0:55733:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44225bbf8 00010000:00010000:1.0:1787826425.773463:0:55733:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826425.773538:0:55733:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826425.773560:0:55733:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826425.773570:0:55733:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=27 00000080:00200000:1.0:1787826425.773576:0:55733:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826425.773641:0:55733:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=267 00000080:00200000:1.0:1787826425.773681:0:55733:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826425.773691:0:55733:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826425.773699:0:55733:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826425.773708:0:55733:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826425.773715:0:55733:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826425.773719:0:55733:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826425.773723:0:55733:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826425.773727:0:55733:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44225bbf8 00010000:00010000:1.0:1787826425.773730:0:55733:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826425.773738:0:55733:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826425.773747:0:55733:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826425.773752:0:55733:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=267 00000080:00200000:1.0:1787826425.773756:0:55733:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 44 00000080:00200008:1.0:1787826425.773763:0:55733:0:(file.c:1090:ll_file_open()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00000080:00200080:1.0:1787826425.773801:0:55733:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8) cmd=c0086915 arg=11023d0 00000100:00100000:1.0:1787826425.774111:0:55733:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160b40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55733:1874670563996928:192.168.203.139@tcp:34:lfs.0 00000100:00100000:1.0:1787826425.774663:0:55733:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3840 going to sleep for 11 seconds 00000100:00100000:3.0:1787826425.782273:0:55733:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0160b40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55733:1874670563996928:192.168.203.139@tcp:34:lfs.0 00000080:00200008:3.0:1787826425.783553:0:55733:0:(file.c:479:ll_file_release()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00010000:00010000:3.0:1787826425.784260:0:55733:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9600/0x8e7c99ee62ab0c1c lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b818c9 expref: -99 pid: 55483 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000080:00200000:3.0:1787826425.831010:0:55735:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826425.831054:0:55735:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826425.831080:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826425.831101:0:55735:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826425.831110:0:55735:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826425.831120:0:55735:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826425.831124:0:55735:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826425.831135:0:55735:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44221bbf8 00010000:00010000:3.0:1787826425.831139:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826425.831150:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826425.831170:0:55735:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826425.831191:0:55735:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:3.0:1787826425.831199:0:55735:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787826425.831204:0:55735:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:3.0:1787826425.831208:0:55735:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 44 00000080:00200008:3.0:1787826425.831215:0:55735:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154400 00000002:00010000:3.0:1787826425.831234:0:55735:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114401 00010000:00010000:3.0:1787826425.831298:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1775 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.831307:0:55735:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1775 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787826425.831321:0:55735:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787826425.831335:0:55735:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff97688cb5d2c0 x1874670563997184/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787826425.831346:0:55735:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1775 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826425.831376:0:55735:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670563997184:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826425.831630:0:55735:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25540 going to sleep for 11 seconds 00000100:00100000:3.0:1787826425.840435:0:55735:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670563997184:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826425.840472:0:55735:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1775 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b837af expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.840493:0:55735:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1775 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b837af expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.840506:0:55735:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1775 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b837af expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826425.840525:0:55735:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff97688cb5d2c0 x1874670563997184/t8589935243(8589935243) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826425.840540:0:55735:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff97688cb5d2c0 x1874670563997184/t8589935243(8589935243) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826425.840554:0:55735:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1775 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b837af expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826425.840571:0:55735:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1775 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b837af expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.840586:0:55735:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826425.840594:0:55735:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826425.840661:0:55735:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 37 -> 37 00000080:00010000:3.0:1787826425.840668:0:55735:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1775 00000080:00010000:3.0:1787826425.840675:0:55735:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1775 from it ffffb2d44221bbf0 00010000:00010000:3.0:1787826425.840680:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1775 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b837af expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.840696:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1775 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b837af expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000002:00100000:3.0:1787826425.840731:0:55735:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff97688cb5d2c0 x1874670563997184/t8589935243(8589935243) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1787826425.841247:0:55735:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:3.0:1787826425.841259:0:55735:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:3.0:1787826425.841364:0:55735:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826425.841374:0:55735:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:3.0:1787826425.841403:0:55735:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826425.841418:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826425.841432:0:55735:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826425.841440:0:55735:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826425.841446:0:55735:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:3.0:1787826425.841451:0:55735:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1767 00000080:00010000:3.0:1787826425.841455:0:55735:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1767 from it ffffb2d44221bcd8 00010000:00010000:3.0:1787826425.841459:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826425.841468:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826425.841495:0:55735:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1787826425.841509:0:55735:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:3.0:1787826425.841532:0:55735:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1787826425.841536:0:55735:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff9768837ffe30, osc lock ffff9768984ef998, flags 0x400001002 00010000:00010000:3.0:1787826425.841556:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826425.841569:0:55735:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826425.841579:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826425.841588:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826425.841632:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab177c lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.841642:0:55735:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab177c lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826425.841744:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563997312:192.168.203.139@tcp:101:lfs.0 00000020:00200000:3.0:1787826425.841926:0:55735:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1787826425.847194:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826425.849215:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585290112 00000100:00100000:0.0:1787826425.849236:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585290112 00000100:00100000:1.0:1787826425.853739:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 594 00000100:00100000:1.0:1787826425.853756:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b01621c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:30696:x1874670585290112:12345-192.168.203.139@tcp:106: 00010000:00010000:1.0:1787826425.853779:0:42211:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826425.853795:0:42211:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00010000:1.0:1787826425.853846:0:42211:0:(cl_object.c:322:cl_object_glimpse()) header@000000003787b4ec[0x110, 1, [0x200000405:0x40:0x0] hash] 00000020:00010000:1.0:1787826425.853849:0:42211:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826425 blocks: 0 00000020:00010000:1.0:1787826425.853854:0:42211:0:(cl_object.c:322:cl_object_glimpse()) header@000000008b99b3d0[0x116, 2, [0x2c0000401:0x10e:0x0]] 00000020:00010000:1.0:1787826425.853856:0:42211:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826425 blocks: 0 00000100:00100000:1.0:1787826425.853943:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b01621c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:30696:x1874670585290112:12345-192.168.203.139@tcp:106: Request processed in 190us (6770us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826425.853951:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 594 00010000:00010000:2.0:1787826425.859881:0:2404:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab177c lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.860048:0:2404:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab177c lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.860064:0:2404:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab177c lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.860073:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab177c lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.860083:0:2404:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab177c lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787826425.860099:0:2404:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:2.0:1787826425.860109:0:2404:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1787826425.860129:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab177c lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.860137:0:2404:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab177c lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.860145:0:2404:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab177c lrc: 4/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.860153:0:2404:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976888fb8a00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:2.0:1787826425.860164:0:2404:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab177c lrc: 5/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.860178:0:2404:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1787826425.860196:0:2404:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab177c lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.860204:0:2404:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000afb0db32/0x8e7c99ee62ab177c lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826425.860238:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563997312:192.168.203.139@tcp:101:lfs.0 00000080:00200000:3.0:1787826425.860312:0:55735:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:3.0:1787826425.860320:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf04f0 (ffff976890d10128/0) at cl_lock_release():245 00000020:00010000:3.0:1787826425.860323:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf04f0 (ffff976890d10128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826425.860331:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768837ffe30 (ffff976883b9d518/1) at cl_lock_cancel():142 00000020:00010000:3.0:1787826425.860334:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf04f0 (ffff976890d10128/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826425.860336:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768837ffe30 (ffff976890d10128/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826425.860350:0:55735:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 37 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1787826425.860441:0:55735:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154000 00010000:00010000:3.0:1787826425.860476:0:55735:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1775 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b837af expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826425.860640:0:55735:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826425.860662:0:55735:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826425.860675:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826425.860683:0:55735:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826425.860690:0:55735:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826425.860699:0:55735:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826425.860702:0:55735:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826425.860709:0:55735:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44221bbf8 00010000:00010000:3.0:1787826425.860712:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826425.860722:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826425.860740:0:55735:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826425.860748:0:55735:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:3.0:1787826425.860755:0:55735:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787826425.860759:0:55735:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:3.0:1787826425.860764:0:55735:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 46 00000080:00200008:3.0:1787826425.860773:0:55735:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 1154402 00000002:00010000:3.0:1787826425.860790:0:55735:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114403 00010000:00010000:3.0:1787826425.860897:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab1783 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.860904:0:55735:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab1783 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787826425.860916:0:55735:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787826425.860926:0:55735:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff97688cb5da40 x1874670563997568/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787826425.860935:0:55735:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab1783 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826425.860960:0:55735:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5da40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670563997568:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826425.861202:0:55735:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25900 going to sleep for 11 seconds 00000100:00100000:1.0:1787826425.867928:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826425.867991:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585290240 00000100:00100000:0.0:1787826425.868003:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585290240 00000100:00100000:0.0:1787826425.868079:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 595 00000100:00100000:0.0:1787826425.868091:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0163840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6520:x1874670585290240:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826425.869722:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826425.869951:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826425.869965:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0163840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6520:x1874670585290240:12345-192.168.203.139@tcp:104: Request processed in 1871us (2047us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826425.869978:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 595 00010000:00010000:2.0:1787826425.874680:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826425.874702:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826425.874717:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976890fde600 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:2.0:1787826425.874724:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826425.874735:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826425.874746:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826425.874781:0:42220:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1775 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b837af expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826425.874883:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 2/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826425.874891:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1787826425.874997:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563997696:192.168.203.139@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1787826425.875880:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 1/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826425.875895:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde600/0x8e7c99ee62ab1767 lrc: 0/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83785 expref: -99 pid: 55729 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826425.875917:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826425.875920:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826425.875922:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826425.880168:0:55735:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5da40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670563997568:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826425.880212:0:55735:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode CW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab1783 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b837cb expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.880227:0:55735:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab1783 lrc: 4/1,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b837cb expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.880243:0:55735:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab1783 lrc: 4/1,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b837cb expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.880255:0:55735:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab1783 lrc: 4/1,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b837cb expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.880272:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab1783 lrc: 5/1,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b837cb expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.880292:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab1783 lrc: 5/1,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b837cb expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.880300:0:55735:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab1783 lrc: 4/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b837cb expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826425.880314:0:55735:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff97688cb5da40 x1874670563997568/t8589935244(8589935244) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826425.880329:0:55735:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff97688cb5da40 x1874670563997568/t8589935244(8589935244) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826425.880345:0:55735:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab1783 lrc: 3/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b837cb expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.880663:0:55735:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 8 res 8589935621/29 (0 0) 00000080:00200000:0.0:1787826425.880672:0:55735:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1787826425.880734:0:55735:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1783 00000080:00010000:0.0:1787826425.880744:0:55735:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1783 from it ffffb2d44221bbf0 00010000:00010000:0.0:1787826425.880748:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab1783 lrc: 3/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b837cb expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.880758:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab1783 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b837cb expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:0.0:1787826425.880793:0:55735:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff97688cb5da40 x1874670563997568/t8589935244(8589935244) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:0.0:1787826425.880939:0:55735:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=800466af arg=7fff830a9d8c 00000080:00200000:0.0:1787826425.881058:0:55735:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826425.881078:0:55735:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826425.881092:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826425.881478:0:55735:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826425.881498:0:55735:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826425.881506:0:55735:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826425.881510:0:55735:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826425.881516:0:55735:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44221bbf8 00010000:00010000:0.0:1787826425.881519:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826425.881528:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826425.881759:0:55735:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826425.881770:0:55735:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=24 00000080:00200000:0.0:1787826425.881777:0:55735:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826425.881797:0:55735:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 3 00000080:00200000:0.0:1787826425.881801:0:55735:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=. :VOLATILE:0000:4206CB4A:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), file ffff9768904a0600, open_flags 2f9c1, mode 8180 opened 0 00000080:00200000:0.0:1787826425.881807:0:55735:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=. :VOLATILE:0000:4206CB4A:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=open|creat 00000002:00010000:0.0:1787826425.881970:0:55735:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: . :VOLATILE:0000:4206CB4A:fd=03,[0x200000405:0x42:0x0]) in obj [0x200000bd0:0x4:0x0], intent: open|creat flags 0574702 00010000:00010000:0.0:1787826425.882037:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab178a lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.882046:0:55735:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab178a lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826425.882064:0:55735:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826425.882075:0:55735:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aec780 x1874670563997952/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/66640 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826425.882084:0:55735:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab178a lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826425.882213:0:55735:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670563997952:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826425.882754:0:55735:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e780 going to sleep for 11 seconds 00000100:00100000:3.0:1787826425.887787:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563997696:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826425.888057:0:55735:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670563997952:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826425.888087:0:55735:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab178a lrc: 4/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b837e0 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.888266:0:55735:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab178a lrc: 4/0,1 mode: --/PR res: [0x200000405:0x42:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b837e0 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.888592:0:55735:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab178a lrc: 4/0,1 mode: --/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b837e0 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.888609:0:55735:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab178a lrc: 4/0,1 mode: PR/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b837e0 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.888623:0:55735:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab178a lrc: 4/0,1 mode: PR/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b837e0 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.888639:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab178a lrc: 5/1,1 mode: PR/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b837e0 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.888654:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab178a lrc: 5/1,1 mode: PR/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b837e0 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.888663:0:55735:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab178a lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b837e0 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826425.888676:0:55735:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff976882aec780 x1874670563997952/t8589935245(8589935245) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826425.888689:0:55735:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aec780 x1874670563997952/t8589935245(8589935245) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826425.888701:0:55735:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab178a lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b837e0 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.888713:0:55735:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/66 (0 0) 00000080:00200000:0.0:1787826425.888719:0:55735:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd0:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826425.888737:0:55735:0:(namei.c:1628:ll_create_it()) VFS Op:name=. :VOLATILE:0000:4206CB4A:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=open|creat 00000080:00200000:0.0:1787826425.888780:0:55735:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000405:0x42:0x0](ffff9768901aba88) 00000080:00200000:0.0:1787826425.888788:0:55735:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x42:0x0], updating i_size 0 00000080:00200000:0.0:1787826425.889072:0:55735:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x42:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:0.0:1787826425.889079:0:55735:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000405:0x42:0x0](ffff9768901aba88) 00000080:00010000:0.0:1787826425.889083:0:55735:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x42:0x0](ffff9768901aba88) for lock 0x8e7c99ee62ab178a 00000080:00010000:0.0:1787826425.889095:0:55735:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x42:0x0](ffff9768901aba88) 00000080:00200008:0.0:1787826425.889246:0:55735:0:(file.c:1090:ll_file_open()) START file . :VOLATILE:0000:4206CB4A:fd=03:[0x200000405:0x42:0x0](ffff9768901aba88), flags 574701 00000002:00100000:0.0:1787826425.889275:0:55735:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aec780 x1874670563997952/t8589935245(8589935245) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787826425.889758:0:55735:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab178a from it ffff976886816cc0 00010000:00010000:0.0:1787826425.889765:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab178a lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b837e0 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.889776:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab178a lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b837e0 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826425.889814:0:55735:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x42:0x0](ffff9768901aba88), inode mode 8180 mask 40 00000080:00200000:0.0:1787826425.890088:0:55735:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x42:0x0](ffff9768901aba88), inode mode 8180 mask 2 00000080:00200000:0.0:1787826425.890170:0:55735:0:(xattr.c:414:ll_xattr_set()) VFS Op:inode=[0x200000405:0x42:0x0](ffff9768901aba88), xattr lov 00000002:00010000:0.0:1787826425.890208:0:55735:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x42:0x0]) in obj [0x200000405:0x42:0x0], intent: open flags 040000002 00010000:00010000:0.0:1787826425.890346:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1791 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x42:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.890471:0:55735:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1791 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x42:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826425.890545:0:55735:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826425.890558:0:55735:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aee580 x1874670563998208/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826425.890568:0:55735:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1791 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x42:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826425.890595:0:55735:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670563998208:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826425.891869:0:55735:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e780 going to sleep for 11 seconds 00000100:00100000:1.0:1787826425.894690:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826425.894880:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585290368 00000100:00100000:0.0:1787826425.894893:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585290368 00000100:00100000:0.0:1787826425.894959:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 596 00000100:00100000:0.0:1787826425.894970:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0160000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:16564:x1874670585290368:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826425.894995:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab178a lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b837e0 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826425.895082:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826425.895090:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0160000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:16564:x1874670585290368:12345-192.168.203.139@tcp:104: Request processed in 121us (416us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826425.895109:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 596 00010000:00010000:2.0:1787826425.895723:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826425.895735:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab178a lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b837e0 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.895746:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925dc00 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:2.0:1787826425.895752:0:42221:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab178a lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b837e0 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.895762:0:42221:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab178a lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b837e0 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.895771:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab178a lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b837e0 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826425.895801:0:42221:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x42:0x0]: losing layout lock 00010000:00010000:2.0:1787826425.895891:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab178a lrc: 2/0,0 mode: --/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b837e0 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.895899:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826425.895930:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab178a lrc: 1/0,0 mode: --/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b837e0 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.895940:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925dc00/0x8e7c99ee62ab178a lrc: 0/0,0 mode: --/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b837e0 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.895961:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826425.895964:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826425.895965:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826425.895975:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1787826425.895995:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563998336:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826425.902287:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563998336:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826425.902920:0:55735:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670563998208:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826425.902945:0:55735:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1791 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x42:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.902961:0:55735:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1791 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x42:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.902969:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1791 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x42:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.902977:0:55735:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1791 lrc: 3/0,0 mode: --/CR res: [0x200000405:0x42:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.902986:0:55735:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1791 lrc: 4/0,0 mode: --/CR res: [0x200000405:0x42:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.902993:0:55735:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925c400 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:0.0:1787826425.902997:0:55735:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1791 lrc: 4/0,0 mode: --/CR res: [0x200000405:0x42:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.903010:0:55735:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1791 lrc: 4/0,0 mode: --/CR res: [0x200000405:0x42:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.903018:0:55735:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1791 lrc: 5/0,0 mode: --/CR res: [0x200000405:0x42:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.903026:0:55735:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787826425.903034:0:55735:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1791 lrc: 3/0,0 mode: --/CR res: [0x200000405:0x42:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.903041:0:55735:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1791 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x42:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826425.903073:0:55735:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x800002b, status=0 req@ffff976882aee580 x1874670563998208/t8589935246(8589935246) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826425.903086:0:55735:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aee580 x1874670563998208/t8589935246(8589935246) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826425.903120:0:55735:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x42:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826425.903196:0:55735:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x42:0x0] ignore/verify layout 1/0, layout version 31 restore needed 0 00000080:00200000:0.0:1787826425.903208:0:55735:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x42:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826425.903267:0:55735:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x42:0x0], updating i_size 0 00000002:00100000:0.0:1787826425.903304:0:55735:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aee580 x1874670563998208/t8589935246(8589935246) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826425.903683:0:55735:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976882aee580 x1874670563998208/t8589935246(8589935246) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826425.903696:0:55735:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826425.903702:0:55735:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aed2c0 x1874670563998592/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826425.903731:0:55735:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed2c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670563998592:192.168.203.139@tcp:35:lfs.0 00000100:00100000:0.0:1787826425.903913:0:55735:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e480 going to sleep for 11 seconds 00000100:00100000:0.0:1787826425.911148:0:55735:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed2c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670563998592:192.168.203.139@tcp:35:lfs.0 00000002:00100000:0.0:1787826425.911207:0:55735:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976882aee580 x1874670563998208/t8589935246(8589935246) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826441 ref 2 fl Complete:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1787826425.911258:0:55735:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826425.911268:0:55735:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:0.0:1787826425.911296:0:55735:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826425.911315:0:55735:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 30 res 8589935621/29 (0 0) 00010000:00010000:0.0:1787826425.911390:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.911401:0:55735:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.911411:0:55735:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826425.911433:0:55735:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aecb40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670563998720:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826425.911532:0:55735:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e480 going to sleep for 11 seconds 00000100:00100000:1.0:1787826425.915731:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826425.915984:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585290496 00000100:00100000:0.0:1787826425.915996:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585290496 00000100:00100000:0.0:1787826425.916104:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 597 00000100:00100000:0.0:1787826425.916116:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0162940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:9966:x1874670585290496:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826425.916138:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab1783 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x22153cd396b837cb expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826425.916249:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826425.916256:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0162940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:9966:x1874670585290496:12345-192.168.203.139@tcp:104: Request processed in 144us (675us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826425.916264:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 597 00010000:00010000:3.0:1787826425.916326:0:49796:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826425.916336:0:49796:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab1783 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x22153cd396b837cb expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.916345:0:49796:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976888fb8200 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826425.916351:0:49796:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab1783 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x22153cd396b837cb expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.916375:0:49796:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab1783 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x22153cd396b837cb expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.916383:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab1783 lrc: 3/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x22153cd396b837cb expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.916420:0:49796:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1775 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b837af expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826425.916467:0:49796:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab1783 lrc: 2/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x22153cd396b837cb expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.916474:0:49796:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826425.916515:0:49796:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab1783 lrc: 1/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x22153cd396b837cb expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.916522:0:49796:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab1783 lrc: 0/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x22153cd396b837cb expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826425.916541:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826425.916543:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826425.916545:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826425.916554:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787826425.916574:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563998848:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826425.922871:0:55735:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aecb40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670563998720:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826425.922910:0:55735:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.922924:0:55735:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.922939:0:55735:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.922950:0:55735:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.922964:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.922973:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.922981:0:55735:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826425.922999:0:55735:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff976882aecb40 x1874670563998720/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 576/720 e 0 to 0 dl 1787826441 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826425.923011:0:55735:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826425.923025:0:55735:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.923039:0:55735:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/29 (0 0) 00000080:00200000:0.0:1787826425.923046:0:55735:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826425.923120:0:55735:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 37 -> 37 00000080:00010000:0.0:1787826425.923126:0:55735:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1798 00000080:00010000:0.0:1787826425.923132:0:55735:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:0.0:1787826425.923136:0:55735:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1798 from it ffffb2d44221bcd8 00010000:00010000:0.0:1787826425.923139:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.923149:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826425.923199:0:55735:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787826425.923212:0:55735:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:0.0:1787826425.923237:0:55735:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826425.923241:0:55735:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e55130, osc lock ffff9768849a29c0, flags 0x400001002 00010000:00010000:0.0:1787826425.923261:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826425.923293:0:55735:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826425.923303:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826425.923309:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826425.923366:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab179f lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.923375:0:55735:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab179f lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826425.923412:0:55735:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1787826425.923426:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1787826425.923445:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aecb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563998976:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826425.927040:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563998848:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826425.927786:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826425.928042:0:53061:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:0.0:1787826425.928065:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585290624 00000100:00100000:0.0:1787826425.928074:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585290624 00000100:00100000:0.0:1787826425.928245:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 598 00000100:00100000:0.0:1787826425.928257:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff97688a33f050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:30695:x1874670585290624:12345-192.168.203.139@tcp:106: 00010000:00010000:0.0:1787826425.928287:0:53061:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826425.928304:0:53061:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00000020:00010000:0.0:1787826425.928337:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@000000003787b4ec[0x110, 1, [0x200000405:0x40:0x0] hash] 00000020:00010000:0.0:1787826425.928340:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826425 blocks: 0 00000020:00010000:0.0:1787826425.928344:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@000000008b99b3d0[0x116, 2, [0x2c0000401:0x10e:0x0]] 00000020:00010000:0.0:1787826425.928346:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826425 blocks: 0 00000100:00100000:0.0:1787826425.928431:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff97688a33f050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:30695:x1874670585290624:12345-192.168.203.139@tcp:106: Request processed in 175us (649us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826425.928440:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 598 00010000:00010000:0.0:1787826425.931472:0:2402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab179f lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.931632:0:2402:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab179f lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.931647:0:2402:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab179f lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.931657:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab179f lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.931666:0:2402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab179f lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826425.931688:0:2402:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:0.0:1787826425.931696:0:2402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1787826425.931714:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab179f lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.931723:0:2402:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab179f lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.931733:0:2402:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab179f lrc: 4/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.931740:0:2402:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925cc00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:0.0:1787826425.931746:0:2402:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab179f lrc: 5/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.931758:0:2402:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787826425.931765:0:2402:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab179f lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.931772:0:2402:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000009e048b4e/0x8e7c99ee62ab179f lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826425.931796:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aecb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670563998976:192.168.203.139@tcp:101:lfs.0 00000080:00200000:3.0:1787826425.931993:0:55735:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:3.0:1787826425.932001:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf3160 (ffff976890d106c8/0) at cl_lock_release():245 00000020:00010000:3.0:1787826425.932004:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf3160 (ffff976890d106c8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826425.932011:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e55130 (ffff976890d10e18/1) at cl_lock_cancel():142 00000020:00010000:3.0:1787826425.932015:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf3160 (ffff976890d106c8/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826425.932018:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e55130 (ffff976890d106c8/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826425.932032:0:55735:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 37 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1787826425.932059:0:55735:0:(file.c:6313:ll_getattr_dentry()) START file . :VOLATILE:0000:4206CB4A:fd=03:[0x200000405:0x42:0x0](ffff9768901aba88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826425.932068:0:55735:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x42:0x0](ffff9768901aba88),name=. :VOLATILE:0000:4206CB4A:fd=03 00000002:00010000:3.0:1787826425.932095:0:55735:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x42:0x0]) in obj [0x200000405:0x42:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826425.932251:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab17a6 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x42:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.932261:0:55735:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab17a6 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x42:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.932272:0:55735:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab17a6 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x42:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826425.932309:0:55735:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670563999104:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826425.932759:0:55735:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25900 going to sleep for 11 seconds 00000100:00100000:3.0:1787826425.937067:0:55735:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670563999104:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826425.937169:0:55735:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab17a6 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x42:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8381f expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.937194:0:55735:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab17a6 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8381f expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.937209:0:55735:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab17a6 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8381f expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.937224:0:55735:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab17a6 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8381f expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.937239:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab17a6 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8381f expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.937248:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab17a6 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8381f expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.937257:0:55735:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab17a6 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8381f expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826425.937271:0:55735:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff97688cb5cf00 x1874670563999104/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 576/720 e 0 to 0 dl 1787826441 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826425.937465:0:55735:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab17a6 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8381f expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787826425.937985:0:55735:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab17a6 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8381f expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.938003:0:55735:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/66 (0 0) 00000080:00200000:2.0:1787826425.938011:0:55735:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000405:0x42:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787826425.938079:0:55735:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x42:0x0]: layout version change: 4294967294 -> 1 00000080:00010000:2.0:1787826425.938086:0:55735:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x42:0x0](ffff9768901aba88) for lock 0x8e7c99ee62ab17a6 00000080:00010000:2.0:1787826425.938091:0:55735:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x42:0x0](ffff9768901aba88) 00000080:00010000:2.0:1787826425.938094:0:55735:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab17a6 from it ffffb2d44221bcd8 00010000:00010000:2.0:1787826425.938162:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab17a6 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b8381f expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.938173:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab17a6 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8381f expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826425.938226:0:55735:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x42:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787826425.938239:0:55735:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x42:0x0] 00000008:00010000:2.0:1787826425.938260:0:55735:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787826425.938263:0:55735:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff97688757cc30, osc lock ffff976883f7b4b8, flags 0x400001002 00010000:00010000:2.0:1787826425.938495:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17ad lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.938508:0:55735:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17ad lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826425.938694:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563999232:192.168.203.139@tcp:101:lfs.0 00000020:00200000:2.0:1787826425.939017:0:55735:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787826425.944805:0:2404:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17ad lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.944962:0:2404:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17ad lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83826 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.944982:0:2404:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17ad lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83826 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.944997:0:2404:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17ad lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83826 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787826425.945016:0:2404:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17ad lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83826 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1787826425.945030:0:2404:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1787826425.945083:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17ad lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83826 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.945094:0:2404:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17ad lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83826 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.945108:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17ad lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83826 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.945118:0:2404:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17ad lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83826 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1787826425.945143:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563999232:192.168.203.139@tcp:101:lfs.0 00000080:00200000:0.0:1787826425.945222:0:55735:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x42:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787826425.945231:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf2018 (ffff976890d105a8/0) at cl_lock_release():245 00000020:00010000:0.0:1787826425.945236:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf2018 (ffff976890d105a8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826425.945242:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757cc30 (ffff976890d102d8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826425.945247:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf2018 (ffff976890d105a8/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826425.945250:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757cc30 (ffff976890d105a8/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826425.945267:0:55735:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x42:0x0] ignore/verify layout 0/1, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200080:0.0:1787826425.945459:0:55735:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=7fff830abed4 00000002:00010000:0.0:1787826425.945492:0:55735:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 01004040000001 00010000:00010000:0.0:1787826425.945552:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab17b4 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.945561:0:55735:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x81000000001000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab17b4 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826425.945573:0:55735:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826425.945581:0:55735:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aecb40 x1874670563999488/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826425.945591:0:55735:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab17b4 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x81000000000000 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826425.945615:0:55735:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aecb40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670563999488:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826425.946137:0:55735:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e900 going to sleep for 11 seconds 00000100:00100000:2.0:1787826425.950271:0:55735:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aecb40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670563999488:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826425.950469:0:55735:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab17b4 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83834 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.950484:0:55735:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab17b4 lrc: 4/1,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83834 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.950499:0:55735:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab17b4 lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83834 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.950511:0:55735:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab17b4 lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83834 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.950527:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab17b4 lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83834 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.950607:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab17b4 lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83834 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.950615:0:55735:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab17b4 lrc: 4/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83834 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826425.950629:0:55735:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x600002b, status=0 req@ffff976882aecb40 x1874670563999488/t8589935248(8589935248) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787826425.950644:0:55735:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aecb40 x1874670563999488/t8589935248(8589935248) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826425.950663:0:55735:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab17b4 lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83834 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.950680:0:55735:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 2 res 8589935621/29 (0 0) 00000080:00200000:2.0:1787826425.950688:0:55735:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000002:00100000:2.0:1787826425.950713:0:55735:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aecb40 x1874670563999488/t8589935248(8589935248) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826441 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1787826425.950722:0:55735:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab17b4 00000080:00010000:2.0:1787826425.950729:0:55735:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab17b4 from it ffffb2d44221bcc0 00010000:00010000:2.0:1787826425.950732:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab17b4 lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83834 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.950740:0:55735:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab17b4 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83834 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200008:2.0:1787826425.950813:0:55735:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787826425.950992:0:55735:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:2.0:1787826425.951011:0:55735:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826425.951022:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.951034:0:55735:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826425.951041:0:55735:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826425.951048:0:55735:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:2.0:1787826425.951051:0:55735:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1798 00000080:00010000:2.0:1787826425.951055:0:55735:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1798 from it ffffb2d44221bcd8 00010000:00010000:2.0:1787826425.951058:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826425.951068:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826425.951099:0:55735:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787826425.951112:0:55735:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:2.0:1787826425.951135:0:55735:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787826425.951139:0:55735:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff97688757cd30, osc lock ffff976883f7ad68, flags 0x400001002 00010000:00010000:2.0:1787826425.951157:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826425.951175:0:55735:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826425.951194:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826425.951202:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826425.951246:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17bb lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.951255:0:55735:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17bb lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826425.951484:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563999616:192.168.203.139@tcp:101:lfs.0 00000020:00200000:2.0:1787826425.951603:0:55735:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1787826425.954520:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826425.954675:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585290752 00000100:00100000:0.0:1787826425.954687:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585290752 00000100:00100000:0.0:1787826425.954782:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 599 00000100:00100000:0.0:1787826425.954792:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0160780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:28853:x1874670585290752:12345-192.168.203.139@tcp:106: 00010000:00010000:0.0:1787826425.954812:0:42210:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826425.954855:0:42210:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00000020:00010000:0.0:1787826425.954888:0:42210:0:(cl_object.c:322:cl_object_glimpse()) header@000000003787b4ec[0x110, 1, [0x200000405:0x40:0x0] hash] 00000020:00010000:0.0:1787826425.954890:0:42210:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826425 blocks: 0 00000020:00010000:0.0:1787826425.954894:0:42210:0:(cl_object.c:322:cl_object_glimpse()) header@000000008b99b3d0[0x116, 2, [0x2c0000401:0x10e:0x0]] 00000020:00010000:0.0:1787826425.954895:0:42210:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826425 blocks: 0 00000100:00100000:0.0:1787826425.954978:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0160780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:28853:x1874670585290752:12345-192.168.203.139@tcp:106: Request processed in 187us (469us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826425.954987:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 599 00010000:00010000:2.0:1787826425.957809:0:2403:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17bb lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.958004:0:2403:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17bb lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.958020:0:2403:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17bb lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.958035:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17bb lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.958044:0:2403:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17bb lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787826425.958061:0:2403:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:2.0:1787826425.958070:0:2403:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1787826425.958136:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17bb lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.958146:0:2403:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17bb lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.958156:0:2403:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17bb lrc: 4/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.958164:0:2403:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515000 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:2.0:1787826425.958171:0:2403:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17bb lrc: 5/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.958237:0:2403:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1787826425.958247:0:2403:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17bb lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.958262:0:2403:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17bb lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826425.958297:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563999616:192.168.203.139@tcp:101:lfs.0 00000080:00200000:0.0:1787826425.958525:0:55735:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787826425.958534:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf09e0 (ffff976890d10cf8/0) at cl_lock_release():245 00000020:00010000:0.0:1787826425.958595:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf09e0 (ffff976890d10cf8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826425.958602:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757cd30 (ffff976890d10488/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826425.958606:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf09e0 (ffff976890d10cf8/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826425.958609:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757cd30 (ffff976890d10cf8/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826425.958624:0:55735:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 37 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787826425.958653:0:55735:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826425.958662:0:55735:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:0.0:1787826425.958689:0:55735:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826425.958710:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826425.958725:0:55735:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826425.958740:0:55735:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826425.958751:0:55735:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:0.0:1787826425.958756:0:55735:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1798 00000080:00010000:0.0:1787826425.958763:0:55735:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1798 from it ffffb2d44221bcd8 00010000:00010000:0.0:1787826425.958767:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826425.958778:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826425.958804:0:55735:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 37 restore needed 0 00000001:00010000:0.0:1787826425.958815:0:55735:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:2.0:1787826425.959113:0:55735:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787826425.959118:0:55735:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff97688757c730, osc lock ffff976883f7bc08, flags 0x400001002 00010000:00010000:2.0:1787826425.959136:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826425.959147:0:55735:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826425.959159:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826425.959168:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1787826425.959224:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab17c2 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826425.959233:0:55735:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab17c2 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826425.959329:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563999744:192.168.203.139@tcp:101:lfs.0 00000020:00200000:2.0:1787826425.959738:0:55735:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1787826425.962664:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826425.962786:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585290880 00000100:00100000:0.0:1787826425.962799:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585290880 00000100:00100000:0.0:1787826425.963088:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 600 00000100:00100000:0.0:1787826425.963102:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0160f00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8435:x1874670585290880:12345-192.168.203.139@tcp:106: 00010000:00010000:0.0:1787826425.963124:0:53061:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826425.963140:0:53061:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00000020:00010000:0.0:1787826425.963174:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@000000003787b4ec[0x110, 1, [0x200000405:0x40:0x0] hash] 00000020:00010000:0.0:1787826425.963177:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826425 blocks: 0 00000020:00010000:0.0:1787826425.963192:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@000000008b99b3d0[0x116, 2, [0x2c0000401:0x10e:0x0]] 00000020:00010000:0.0:1787826425.963195:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826425 blocks: 0 00000100:00100000:0.0:1787826425.963268:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0160f00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8435:x1874670585290880:12345-192.168.203.139@tcp:106: Request processed in 170us (618us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826425.963350:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 600 00010000:00010000:3.0:1787826425.966672:0:2404:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab17c2 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.966702:0:2404:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab17c2 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.966721:0:2404:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab17c2 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.966730:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab17c2 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.966739:0:2404:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab17c2 lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826425.966757:0:2404:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:3.0:1787826425.966765:0:2404:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826425.967156:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab17c2 lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.967167:0:2404:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab17c2 lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.967176:0:2404:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab17c2 lrc: 4/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.967193:0:2404:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515800 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826425.967200:0:2404:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab17c2 lrc: 5/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.967221:0:2404:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826425.967230:0:2404:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab17c2 lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826425.967240:0:2404:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab17c2 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826425.967265:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670563999744:192.168.203.139@tcp:101:lfs.0 00000080:00200000:2.0:1787826425.967497:0:55735:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1787826425.967505:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf09e0 (ffff976890d10cf8/0) at cl_lock_release():245 00000020:00010000:2.0:1787826425.967509:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf09e0 (ffff976890d10cf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787826425.967515:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757c730 (ffff976890d10d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787826425.967519:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf09e0 (ffff976890d10cf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1787826425.967523:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757c730 (ffff976890d10cf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1787826425.967593:0:55735:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 37 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200080:2.0:1787826425.967630:0:55735:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=801066da arg=7fff830abdd0 00000080:00200000:2.0:1787826425.967644:0:55735:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 37 restore needed 0 00020000:00200000:2.0:1787826425.967661:0:55735:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1787826425.967784:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563999872:192.168.203.139@tcp:1:lfs.0 00000100:00100000:1.0:1787826425.971104:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826425.971214:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585291008 00000100:00100000:0.0:1787826425.971225:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585291008 00000100:00100000:0.0:1787826425.971383:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 601 00000100:00100000:0.0:1787826425.971396:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b01621c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:14864:x1874670585291008:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826425.971418:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826425.971516:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826425.971524:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b01621c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:14864:x1874670585291008:12345-192.168.203.139@tcp:104: Request processed in 131us (435us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826425.971532:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 601 00010000:00010000:2.0:1787826425.971702:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826425.971712:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1787826425.971723:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925d000 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:2.0:1787826425.971730:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400020000 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1787826425.971874:0:42220:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x40:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787826425.971892:0:42220:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x40:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1787826425.971956:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 2/0,0 mode: --/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1787826425.971965:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826425.971993:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 1/0,0 mode: --/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1787826425.972003:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000045cdfb28/0x8e7c99ee62ab171a lrc: 0/0,0 mode: --/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x22153cd396b8367b expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1787826425.972030:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826425.972033:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826425.972034:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826425.972247:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564000000:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826425.977354:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670563999872:192.168.203.139@tcp:1:lfs.0 00000080:00200000:0.0:1787826425.977861:0:55735:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:0.0:1787826425.977874:0:55735:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:0.0:1787826425.977970:0:55735:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826425.977980:0:55735:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:0.0:1787826425.978007:0:55735:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826425.978033:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826425.978051:0:55735:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826425.978061:0:55735:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826425.978072:0:55735:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:0.0:1787826425.978076:0:55735:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1798 00000080:00010000:0.0:1787826425.978087:0:55735:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1798 from it ffffb2d44221bcd8 00010000:00010000:0.0:1787826425.978091:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826425.978100:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1787826425.978131:0:55735:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787826425.978143:0:55735:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:0.0:1787826425.978166:0:55735:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826425.978170:0:55735:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e55b30, osc lock ffff9768849a2270, flags 0x400001002 00010000:00010000:0.0:1787826425.978241:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.978252:0:55735:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826425.978305:0:55735:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1787826425.978563:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670564000128:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826425.981147:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564000000:192.168.203.139@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1787826425.981738:0:2402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.981764:0:2402:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.981782:0:2402:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.981802:0:2402:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826425.981819:0:2402:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826425.981851:0:2402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1787826425.981872:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.981882:0:2402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.981890:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826425.981970:0:2402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1787826425.981992:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670564000128:192.168.203.139@tcp:101:lfs.0 00000080:00200000:1.0:1787826425.982080:0:55735:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:1.0:1787826425.982089:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9768ad1fd148 (ffff976890d10d88/0) at cl_lock_release():245 00000020:00010000:1.0:1787826425.982093:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768ad1fd148 (ffff976890d10d88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1787826425.982099:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e55b30 (ffff976890d10488/1) at cl_lock_cancel():142 00000020:00010000:1.0:1787826425.982103:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768ad1fd148 (ffff976890d10d88/0) at cl_lock_fini():72 00000020:00010000:1.0:1787826425.982106:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e55b30 (ffff976890d10d88/1) at cl_lock_fini():72 00000080:00200000:1.0:1787826425.982152:0:55735:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 37 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1787826426.050276:0:55735:0:(file.c:5224:ll_file_seek()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0], offset: 0, type: (null) 00000080:00200000:1.0:1787826426.050464:0:55735:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 33 restore needed 0 00000001:00010000:1.0:1787826426.050479:0:55735:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:1.0:1787826426.050511:0:55735:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1787826426.050516:0:55735:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e79a30, osc lock ffff976883ff8ea0, flags 0x400001002 00010000:00010000:1.0:1787826426.050548:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826426.050579:0:55735:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:1.0:1787826426.050596:0:55735:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1787826426.050600:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826426.050611:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:1.0:1787826426.050631:0:55735:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1787826426.050642:0:55735:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:1.0:1787826426.050648:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976890e938c8 (ffff97688386c998/0) at cl_lock_release():245 00000020:00010000:1.0:1787826426.050652:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976890e938c8 (ffff97688386c998/0) at cl_lock_cancel():142 00000020:00010000:1.0:1787826426.050655:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e79a30 (ffff976890d10908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1787826426.050661:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976890e938c8 (ffff97688386c998/0) at cl_lock_fini():72 00000020:00010000:1.0:1787826426.050664:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e79a30 (ffff97688386c998/1) at cl_lock_fini():72 00000080:00200000:1.0:1787826426.050682:0:55735:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 37 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1787826426.050691:0:55735:0:(file.c:5262:ll_file_seek()) COMPLETED file f60b.sanity-flr:[0x200000405:0x1d:0x0], offset: 0, type: (null), rc = 0 00000080:00200008:1.0:1787826426.050697:0:55735:0:(file.c:5224:ll_file_seek()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0], offset: 0, type: SEEK_HOLE 00000080:00200000:1.0:1787826426.050702:0:55735:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 37 restore needed 0 00020000:00200000:1.0:1787826426.050709:0:55735:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1787826426.050728:0:55735:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000080:00200000:1.0:1787826426.050736:0:55735:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1787826426.050742:0:55735:0:(file.c:1848:ll_io_set_mirror()) f60b.sanity-flr: desiginated mirror: 0 00000080:00200000:1.0:1787826426.050745:0:55735:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:1.0:1787826426.050750:0:55735:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1787826426.050754:0:55735:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 18446744073709551615] 00000008:00010000:1.0:1787826426.050760:0:55735:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 8 00000008:00010000:1.0:1787826426.050762:0:55735:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e79a30, osc lock ffff976883ff8ea0, flags 0x0 00010000:00010000:1.0:1787826426.050771:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826426.050780:0:55735:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826426.050789:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826426.050797:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826426.050804:0:55735:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787826426.050813:0:55735:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:1.0:1787826426.050818:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885e73998 (ffff97688386c998/0) at cl_lock_release():245 00000020:00010000:1.0:1787826426.050821:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885e73998 (ffff97688386c998/0) at cl_lock_cancel():142 00000020:00010000:1.0:1787826426.050917:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e79a30 (ffff976890d10908/1) at cl_lock_cancel():142 00010000:00010000:1.0:1787826426.050921:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826426.050936:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00010000:1.0:1787826426.050944:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885e73998 (ffff97688386c998/0) at cl_lock_fini():72 00000020:00010000:1.0:1787826426.050946:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e79a30 (ffff97688386c998/1) at cl_lock_fini():72 00000080:00200000:1.0:1787826426.050952:0:55735:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 37 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1787826426.050955:0:55735:0:(file.c:5262:ll_file_seek()) COMPLETED file f60b.sanity-flr:[0x200000405:0x1d:0x0], offset: 0, type: SEEK_HOLE, rc = -6 00000080:00200080:1.0:1787826426.050974:0:55735:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=66f4 arg=55b 00000080:00200008:1.0:1787826426.050994:0:55735:0:(file.c:2485:do_file_read_iter()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0], ppos: 0, count: 67108864 00000080:00200000:1.0:1787826426.051006:0:55735:0:(file.c:2090:ll_file_io_generic()) f60b.sanity-flr: read ppos: 0, bytes: 67108864 00000080:00200000:1.0:1787826426.051015:0:55735:0:(file.c:1848:ll_io_set_mirror()) f60b.sanity-flr: desiginated mirror: 0 00000020:00200000:1.0:1787826426.051021:0:55735:0:(cl_io.c:218:cl_io_rw_init()) header@00000000a0ca5d5e[0x110, 2, [0x200000405:0x1d:0x0] hash] 00000020:00200000:1.0:1787826426.051023:0:55735:0:(cl_io.c:218:cl_io_rw_init()) io range: 1 [0, 67108864) 2048 0 00000080:00200000:1.0:1787826426.051026:0:55735:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 37 restore needed 0 00000080:00200000:1.0:1787826426.051035:0:55735:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 16383] 00020000:00200000:1.0:1787826426.051042:0:55735:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 67108864) 67108864, 67108864 00020000:00200000:1.0:1787826426.051045:0:55735:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1787826426.051056:0:55735:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 16383] 00000008:00010000:1.0:1787826426.051062:0:55735:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 11 00000008:00010000:1.0:1787826426.051065:0:55735:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e79a30, osc lock ffff976883ff8ea0, flags 0x40000 00000080:00200000:1.0:1787826426.051476:0:55735:0:(vvp_io.c:874:vvp_io_read_start()) f60b.sanity-flr: read [0, 67108864) 00000001:00010000:1.0:1787826426.051488:0:55735:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:1.0:1787826426.051497:0:55735:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1787826426.051499:0:55735:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e79430, osc lock ffff976883ff8750, flags 0x400001002 00010000:00010000:1.0:1787826426.051512:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826426.051528:0:55735:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00200000:1.0:1787826426.051538:0:55735:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1787826426.051546:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826426.051555:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00000020:00200000:1.0:1787826426.051565:0:55735:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1787826426.051570:0:55735:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:1.0:1787826426.051574:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976890e938c8 (ffff97688386c998/0) at cl_lock_release():245 00000020:00010000:1.0:1787826426.051577:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976890e938c8 (ffff97688386c998/0) at cl_lock_cancel():142 00000020:00010000:1.0:1787826426.051579:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e79430 (ffff976890d10908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1787826426.051582:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976890e938c8 (ffff97688386c998/0) at cl_lock_fini():72 00000020:00010000:1.0:1787826426.051584:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e79430 (ffff97688386c998/1) at cl_lock_fini():72 00000080:00200000:1.0:1787826426.051592:0:55735:0:(vvp_io.c:932:vvp_io_read_start()) count:67108864 iocb pos:0 00000080:00200000:1.0:1787826426.051599:0:55735:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), size=67108864 (max 335544320), offset=0=0x0, pages 16384 (max 81920), lockless, parallel 00000020:00010000:1.0:1787826426.051612:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885e73998 (ffff97688386c998/0) at cl_lock_release():245 00000020:00010000:1.0:1787826426.051614:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885e73998 (ffff97688386c998/0) at cl_lock_cancel():142 00000020:00010000:1.0:1787826426.051616:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e79a30 (ffff976890d10908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1787826426.051619:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885e73998 (ffff97688386c998/0) at cl_lock_fini():72 00000020:00010000:1.0:1787826426.051621:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e79a30 (ffff97688386c998/1) at cl_lock_fini():72 00000080:00200000:1.0:1787826426.051626:0:55735:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000020:00200000:1.0:1787826426.051633:0:55735:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1787826426.051637:0:55735:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 16383] 00000080:00200000:1.0:1787826426.051642:0:55735:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 37 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1787826426.051646:0:55735:0:(file.c:2240:ll_file_io_generic()) f60b.sanity-flr: 1 io complete with rc: 0, result: 0, restart: 0 00000020:00200000:1.0:1787826426.051649:0:55735:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1787826426.051654:0:55735:0:(file.c:2326:ll_file_io_generic()) iot: 1, result: 0 00000080:00200008:1.0:1787826426.051668:0:55735:0:(file.c:5401:ll_fsync()) START file: name=. :VOLATILE:0000:4206CB4A:fd=03, fid=[0x200000405:0x42:0x0], start=0, end=9223372036854775807, datasync=0, need_sync 1/0 00000100:00100000:1.0:1787826426.052493:0:55735:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0163840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670564000256:192.168.203.139@tcp:44:lfs.0 00000100:00100000:1.0:1787826426.052748:0:55735:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3f00 going to sleep for 11 seconds 00000100:00100000:1.0:1787826426.060293:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826426.060630:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585291264 00000100:00100000:0.0:1787826426.060643:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585291264 00000100:00100000:0.0:1787826426.060712:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 602 00000100:00100000:0.0:1787826426.060724:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0160000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8436:x1874670585291264:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826426.060748:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000030000 nid: local remote: 0x22153cd396b83689 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826426.060884:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826426.060892:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0160000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8436:x1874670585291264:12345-192.168.203.139@tcp:104: Request processed in 171us (615us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826426.060900:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 602 00010000:00010000:2.0:1787826426.061108:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826426.061117:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000030000 nid: local remote: 0x22153cd396b83689 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.061127:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514800 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:2.0:1787826426.061134:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400030000 nid: local remote: 0x22153cd396b83689 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.061210:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400030020 nid: local remote: 0x22153cd396b83689 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.061219:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826426.061250:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 1/0,0 mode: --/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400030020 nid: local remote: 0x22153cd396b83689 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.061262:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009538a9e4/0x8e7c99ee62ab1721 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400030020 nid: local remote: 0x22153cd396b83689 expref: -99 pid: 55719 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.061308:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.061310:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.061312:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826426.061561:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564000384:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826426.061612:0:55735:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9768877a4800: committing for last_committed 8589935248 gen 1 00000100:00100000:2.0:1787826426.061666:0:55735:0:(client.c:3174:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff97688cb5d2c0 x1874670563997184/t8589935243(8589935243) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826441 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1787826426.061677:0:55735:0:(client.c:3174:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff97688cb5da40 x1874670563997568/t8589935244(8589935244) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826441 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1787826426.061687:0:55735:0:(client.c:3174:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff976882aec780 x1874670563997952/t8589935245(8589935245) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826441 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1787826426.061749:0:55735:0:(client.c:3174:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff976882aecb40 x1874670563999488/t8589935248(8589935248) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826441 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1787826426.061763:0:55735:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0163840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670564000256:192.168.203.139@tcp:44:lfs.0 00000080:00200000:2.0:1787826426.061787:0:55735:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x42:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787826426.061801:0:55735:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1787826426.061921:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826426.061933:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564000512:192.168.203.139@tcp:16:lfs.0 00000100:00100000:3.0:1787826426.071275:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564000384:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826426.071439:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564000512:192.168.203.139@tcp:16:lfs.0 00000080:00200000:1.0:1787826426.071492:0:55735:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x42:0x0] updating i_size 0 i_blocks 0 00000080:00200000:1.0:1787826426.071544:0:55735:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x42:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200080:2.0:1787826426.088336:0:55735:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=801066da arg=7fff830abdd0 00000080:00200000:2.0:1787826426.088417:0:55735:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787826426.088433:0:55735:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1787826426.088598:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564000640:192.168.203.139@tcp:1:lfs.0 00000100:00100000:3.0:1787826426.104278:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564000640:192.168.203.139@tcp:1:lfs.0 00000080:00200008:2.0:1787826426.104508:0:55735:0:(llite_lib.c:2427:ll_setattr_raw()) START file . :VOLATILE:0000:4206CB4A:fd=03:[0x200000405:0x42:0x0](ffff9768901aba88) current size 0, valid attrs 101f0, mode 9768, uid 0, gid 0, new size 4294967296, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826426.000000000 00010000:00010000:2.0:1787826426.104555:0:55735:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab17a6 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b8381f expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826426.104590:0:55735:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x42:0x0]: losing layout lock 00010000:00010000:2.0:1787826426.104651:0:55735:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab17a6 lrc: 1/0,0 mode: --/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b8381f expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.104659:0:55735:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826426.104663:0:55735:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8200/0x8e7c99ee62ab17a6 lrc: 0/0,0 mode: --/PR res: [0x200000405:0x42:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b8381f expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1787826426.104695:0:55735:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787826426.104706:0:55735:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efa580 x1874670564000896/t0(0) o36->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826426.104734:0:55735:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa580 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670564000896:192.168.203.139@tcp:36:lfs.0 00000100:00100000:2.0:1787826426.104891:0:55735:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35900 going to sleep for 11 seconds 00000100:00100000:2.0:1787826426.113418:0:55735:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa580 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670564000896:192.168.203.139@tcp:36:lfs.0 00000080:00200000:2.0:1787826426.113488:0:55735:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x42:0x0] ignore/verify layout 0/0, layout version 37 restore needed 0 00000080:00010000:2.0:1787826426.113524:0:55735:0:(file.c:7076:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000405:0x42:0x0](ffff9768901aba88) 00000002:00010000:2.0:1787826426.113537:0:55735:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x42:0x0]) in obj [0x200000405:0x42:0x0], intent: layout flags 0 00010000:00010000:2.0:1787826426.113684:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17d0 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x42:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.113719:0:55735:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17d0 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x42:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.113732:0:55735:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17d0 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x42:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826426.113757:0:55735:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8f00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670564001024:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826426.114342:0:55735:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35900 going to sleep for 11 seconds 00000100:00100000:2.0:1787826426.117594:0:55735:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8f00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670564001024:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826426.117638:0:55735:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17d0 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x42:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83873 expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.117656:0:55735:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17d0 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x42:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83873 expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826426.117674:0:55735:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff976889ef8f00 x1874670564001024/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 376/480 e 0 to 0 dl 1787826442 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826426.117686:0:55735:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17d0 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x42:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83873 expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787826426.117702:0:55735:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17d0 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x42:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83873 expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.117717:0:55735:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/66 (0 0) 00000080:00200000:2.0:1787826426.117725:0:55735:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000405:0x42:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1787826426.117748:0:55735:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x42:0x0](ffff9768901aba88) for lock 0x8e7c99ee62ab17d0 00000080:00010000:2.0:1787826426.117758:0:55735:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab17d0 from it ffffb2d44221ba78 00010000:00010000:2.0:1787826426.117762:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17d0 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x42:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83873 expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.117773:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17d0 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x42:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83873 expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.117804:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17d0 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x42:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83873 expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.117817:0:55735:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17d0 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x42:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83873 expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1787826426.117869:0:55735:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x42:0x0](000000001d21ca3f) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17d0 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x42:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83873 expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826426.117920:0:55735:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x42:0x0]: layout version change: 4294967294 -> 1 00010000:00010000:2.0:1787826426.117925:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17d0 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x42:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83873 expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.117933:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17d0 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x42:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83873 expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:2.0:1787826426.117960:0:55735:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787826426.117980:0:55735:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:2.0:1787826426.117999:0:55735:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1787826426.118004:0:55735:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff97688757cb30, osc lock ffff976883f7bc08, flags 0x0 00010000:00010000:2.0:1787826426.118019:0:55735:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976889533000 type 11 mode 2 res 10737419265/270 (0 18446744073709551615) 00010000:00010000:2.0:1787826426.118062:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.118071:0:55735:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.118083:0:55735:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826426.118113:0:55735:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8f00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670564001152:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826426.118211:0:55735:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35e40 going to sleep for 11 seconds 00000100:00100000:0.0:1787826426.128947:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826426.132283:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585291392 00000100:00100000:1.0:1787826426.132310:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585291392 00000100:00100000:1.0:1787826426.132662:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 603 00000100:00100000:1.0:1787826426.132675:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aef840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30696:x1874670585291392:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826426.132699:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17ad lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b83826 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826426.132904:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826426.132911:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aef840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30696:x1874670585291392:12345-192.168.203.139@tcp:104: Request processed in 239us (3979us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826426.132919:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 603 00010000:00010000:3.0:1787826426.136619:0:49796:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826426.136642:0:49796:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17ad lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b83826 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826426.136662:0:49796:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514000 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826426.136671:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17ad lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x22153cd396b83826 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826426.136817:0:49796:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x42:0x0] ignore/verify layout 1/0, layout version 1 restore needed 0 00000080:00200000:3.0:1787826426.136855:0:49796:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x42:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1787826426.137166:0:49796:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17ad lrc: 2/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b83826 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.137186:0:49796:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826426.137226:0:49796:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17ad lrc: 1/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b83826 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.137236:0:49796:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17ad lrc: 0/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b83826 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.137255:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.137257:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.137259:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826426.137313:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826426.137339:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564001280:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826426.149625:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:2.0:1787826426.149743:0:55735:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8f00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670564001152:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826426.149781:0:55735:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 4/0,1 mode: --/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.149798:0:55735:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 4/0,1 mode: --/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.149809:0:55735:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 4/0,1 mode: --/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826426.149941:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585291520 00000100:00100000:0.0:1787826426.149949:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585291520 00000100:00100000:0.0:1787826426.150049:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 604 00000100:00100000:0.0:1787826426.150060:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0162940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:6505:x1874670585291520:12345-192.168.203.139@tcp:105: 00010000:00010000:0.0:1787826426.150073:0:42211:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 5/0,1 mode: --/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.150084:0:42211:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 5/0,1 mode: --/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.150094:0:42211:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 5/0,1 mode: --/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.150105:0:42211:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.150115:0:42211:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 6/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.150130:0:42211:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 6/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.150160:0:42211:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff976883515000) 00000100:00100000:0.0:1787826426.150242:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0162940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:6505:x1874670585291520:12345-192.168.203.139@tcp:105: Request processed in 183us (630us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826426.150251:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 604 00010000:00010000:2.0:1787826426.150299:0:55735:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.150310:0:55735:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.150327:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787826426.150341:0:55735:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.150355:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.150373:0:55735:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826426.150587:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564001408:192.168.203.139@tcp:2:lfs.0 00000100:00100000:3.0:1787826426.160289:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564001280:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826426.163460:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564001408:192.168.203.139@tcp:2:lfs.0 00000020:00010000:2.0:1787826426.169553:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885e6f218 (ffff976890d10098/0) at cl_lock_release():245 00000020:00010000:2.0:1787826426.169567:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885e6f218 (ffff976890d10098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787826426.169575:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757cb30 (ffff976890d103f8/1) at cl_lock_cancel():142 00010000:00010000:2.0:1787826426.169588:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.169613:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1787826426.169637:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885e6f218 (ffff976890d10098/0) at cl_lock_fini():72 00000020:00010000:2.0:1787826426.169640:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757cb30 (ffff976890d10098/1) at cl_lock_fini():72 00000080:00200000:2.0:1787826426.169667:0:55735:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x42:0x0] ignore/verify layout 0/0, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1787826426.169685:0:55735:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file . :VOLATILE:0000:4206CB4A:fd=03:[0x200000405:0x42:0x0](ffff9768901aba88) current size 0, valid attrs 100101f0, mode 9768, uid 0, gid 0, new size 4294967296, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826426.000000000, rc 0 00000080:00200008:2.0:1787826426.169770:0:55735:0:(llite_lib.c:2427:ll_setattr_raw()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8) current size 0, valid attrs 101f0, mode 9768, uid 0, gid 0, new size 0, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826426.000000000 00010000:00010000:2.0:1787826426.169818:0:55735:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826426.170084:0:55735:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1775 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b837af expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826426.170152:0:55735:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 1/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826426.170161:0:55735:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826426.170164:0:55735:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1798 lrc: 0/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b8380a expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000020:00100000:2.0:1787826426.170202:0:55735:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787826426.170219:0:55735:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efa1c0 x1874670564001664/t0(0) o36->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826426.170249:0:55735:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa1c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670564001664:192.168.203.139@tcp:36:lfs.0 00000100:00100000:2.0:1787826426.170465:0:55735:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35e40 going to sleep for 11 seconds 00000100:00100000:2.0:1787826426.177290:0:55735:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa1c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670564001664:192.168.203.139@tcp:36:lfs.0 00000080:00200000:2.0:1787826426.177355:0:55735:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 1 restore needed 0 00020000:00200000:2.0:1787826426.177384:0:55735:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787826426.177404:0:55735:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:2.0:1787826426.177424:0:55735:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1787826426.177430:0:55735:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff97688757cb30, osc lock ffff976883f7bc08, flags 0x0 00010000:00010000:2.0:1787826426.177451:0:55735:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9768ad0ae800 type 11 mode 2 res 11811161089/270 (0 18446744073709551615) 00010000:00010000:2.0:1787826426.177510:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.177525:0:55735:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.177535:0:55735:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826426.177561:0:55735:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef9680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670564001792:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826426.177653:0:55735:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35e40 going to sleep for 11 seconds 00000100:00100000:1.0:1787826426.180979:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826426.181098:0:53061:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:0.0:1787826426.181124:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585291648 00000100:00100000:0.0:1787826426.181136:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585291648 00000100:00100000:0.0:1787826426.181208:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 605 00000100:00100000:0.0:1787826426.181219:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768895ba450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8437:x1874670585291648:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826426.181241:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826426.181341:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826426.181348:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768895ba450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8437:x1874670585291648:12345-192.168.203.139@tcp:104: Request processed in 131us (383us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826426.181365:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 605 00010000:00010000:2.0:1787826426.185981:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826426.186000:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826426.186018:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925c800 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:2.0:1787826426.186028:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1787826426.186115:0:42220:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787826426.186134:0:42220:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1787826426.186218:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 2/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826426.186227:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826426.186260:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826426.186268:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000007b53ed2f/0x8e7c99ee62ab17c9 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b83857 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826426.186288:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.186290:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.186292:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826426.186434:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826426.186462:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564001920:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826426.188852:0:55735:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef9680 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670564001792:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826426.188892:0:55735:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 4/0,1 mode: --/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b83888 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.188910:0:55735:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 4/0,1 mode: --/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b83888 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.188920:0:55735:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 4/0,1 mode: --/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b83888 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826426.194494:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826426.194679:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585291776 00000100:00100000:1.0:1787826426.194691:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585291776 00000100:00100000:1.0:1787826426.194989:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 606 00000100:00100000:1.0:1787826426.195002:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aef840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:10340:x1874670585291776:12345-192.168.203.139@tcp:105: 00010000:00010000:1.0:1787826426.195022:0:42211:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 5/0,1 mode: --/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b83888 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.195042:0:42211:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 5/0,1 mode: --/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b83888 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.195053:0:42211:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 5/0,1 mode: --/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b83888 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.195065:0:42211:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x22153cd396b83888 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.195077:0:42211:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 6/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x22153cd396b83888 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.195088:0:42211:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 6/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x22153cd396b83888 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.195241:0:42211:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff976883514000) 00000100:00100000:1.0:1787826426.195330:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aef840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:10340:x1874670585291776:12345-192.168.203.139@tcp:105: Request processed in 330us (850us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826426.195339:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 606 00010000:00010000:2.0:1787826426.196387:0:55735:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b83888 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.196407:0:55735:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b83888 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.196427:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b83888 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787826426.196442:0:55735:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 4/0,2 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x22153cd396b83888 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.196456:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b83888 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.196464:0:55735:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b83888 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826426.198093:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564002048:192.168.203.139@tcp:2:lfs.0 00000100:00100000:3.0:1787826426.198619:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564001920:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826426.203071:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564002048:192.168.203.139@tcp:2:lfs.0 00000020:00010000:2.0:1787826426.203143:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885e6f218 (ffff976890d10098/0) at cl_lock_release():245 00000020:00010000:2.0:1787826426.203149:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885e6f218 (ffff976890d10098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787826426.203156:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757cb30 (ffff976890d103f8/1) at cl_lock_cancel():142 00010000:00010000:2.0:1787826426.203169:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b83888 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.203203:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b83888 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1787826426.203224:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885e6f218 (ffff976890d10098/0) at cl_lock_fini():72 00000020:00010000:2.0:1787826426.203228:0:55735:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757cb30 (ffff976890d10098/1) at cl_lock_fini():72 00000080:00200000:2.0:1787826426.203253:0:55735:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 37 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1787826426.203269:0:55735:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8) current size 0, valid attrs 100101f0, mode 9768, uid 0, gid 0, new size 0, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826426.000000000, rc 0 00000080:00200080:2.0:1787826426.203306:0:55735:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=25f9570 00010000:00010000:2.0:1787826426.203664:0:55735:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab17b4 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0x22153cd396b83834 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.203685:0:55735:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925cc00/0x8e7c99ee62ab17b4 lrc: 0/0,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x85809400000000 nid: local remote: 0x22153cd396b83834 expref: -99 pid: 55735 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826426.203708:0:55735:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976882aecb40 x1874670563999488/t8589935248(8589935248) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826441 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787826426.203720:0:55735:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787826426.203726:0:55735:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889ef8b40 x1874670564002304/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 504/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787826426.203753:0:55735:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8b40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670564002304:192.168.203.139@tcp:35:lfs.0 00000100:00100000:2.0:1787826426.203959:0:55735:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35e40 going to sleep for 11 seconds 00000100:00100000:1.0:1787826426.207114:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826426.207163:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826426.207214:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585291904 00000100:00100000:1.0:1787826426.207225:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585291904 00000100:00100000:1.0:1787826426.207280:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585292032 00000100:00100000:1.0:1787826426.207283:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585292032 00000100:00100000:1.0:1787826426.207303:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 607 00000100:00100000:1.0:1787826426.207313:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0160b40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+6:14257:x1874670585291904:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826426.207336:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1775 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b837af expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1787826426.207449:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826426.207457:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0160b40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+6:14257:x1874670585291904:12345-192.168.203.139@tcp:104: Request processed in 145us (356us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826426.207465:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 607 00000100:00100000:1.0:1787826426.207485:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 608 00000100:00100000:1.0:1787826426.207492:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0163c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585292032:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826426.207500:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83762 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826426.207538:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:1.0:1787826426.207544:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0163c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585292032:12345-192.168.203.139@tcp:104: Request processed in 52us (381us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826426.207550:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 608 00010000:00010000:3.0:1787826426.207627:0:49796:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1787826426.207636:0:49796:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1775 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b837af expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826426.207651:0:49796:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976888fb9a00 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826426.207657:0:49796:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1775 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b837af expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826426.207667:0:49796:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1775 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b837af expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826426.207677:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1775 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 3 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b837af expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826426.207709:0:49796:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83762 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000002:00100000:3.0:1787826426.207769:0:49796:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff97688cb5d2c0 x1874670563997184/t8589935243(8589935243) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826441 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826426.207781:0:49796:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 2 is allocated opc 35, max 7 00000100:00100000:3.0:1787826426.207787:0:49796:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff97688cb5c780 x1874670564002560/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826426.207815:0:49796:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_03:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:49796:1874670564002560:192.168.203.139@tcp:35:ldlm_bl.0 00000100:00100000:3.0:1787826426.207921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25900 going to sleep for 11 seconds 00010000:00010000:2.0:1787826426.208042:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826426.208050:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83762 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.208059:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514200 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:2.0:1787826426.208063:0:42221:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83762 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.208071:0:42221:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83762 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.208078:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b83762 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787826426.208096:0:42221:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x1d:0x0]: losing layout lock 00010000:00010000:2.0:1787826426.208134:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83762 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.208142:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826426.208168:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83762 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.208185:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1760 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83762 expref: -99 pid: 55727 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.208203:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.208205:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.208207:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826426.208321:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826426.208334:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564002688:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826426.211919:0:49796:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_03:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:49796:1874670564002560:192.168.203.139@tcp:35:ldlm_bl.0 00000002:00100000:3.0:1787826426.211963:0:49796:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff97688cb5d2c0 x1874670563997184/t8589935243(8589935243) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826441 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1787826426.212052:0:49796:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1775 lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b837af expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826426.212062:0:49796:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826426.212097:0:49796:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1775 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b837af expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826426.212105:0:49796:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab1775 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b837af expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826426.212131:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826426.212133:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826426.212135:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826426.212145:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787826426.212164:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564002816:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826426.215746:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564002688:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826426.217388:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826426.217553:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585292160 00000100:00100000:1.0:1787826426.217564:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585292160 00000100:00100000:1.0:1787826426.217635:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 609 00000100:00100000:1.0:1787826426.217647:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aec000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585292160:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826426.217671:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17d0 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x42:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83873 expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826426.217766:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826426.217773:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aec000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585292160:12345-192.168.203.139@tcp:104: Request processed in 130us (402us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826426.217782:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 609 00010000:00010000:2.0:1787826426.217994:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826426.218004:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17d0 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x42:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83873 expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.218015:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514800 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:2.0:1787826426.218021:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17d0 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x42:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83873 expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.218030:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17d0 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x42:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83873 expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.218039:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17d0 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x42:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b83873 expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826426.218067:0:42220:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x42:0x0]: losing layout lock 00010000:00010000:2.0:1787826426.218116:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17d0 lrc: 2/0,0 mode: --/CR res: [0x200000405:0x42:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83873 expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.218123:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826426.218143:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17d0 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x42:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83873 expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.218151:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17d0 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x42:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83873 expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.218174:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.218186:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.218188:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826426.218198:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1787826426.218218:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564002944:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826426.218538:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564002816:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826426.221592:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564002944:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826426.232127:0:55735:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8b40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670564002304:192.168.203.139@tcp:35:lfs.0 00000002:00100000:2.0:1787826426.232189:0:55735:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976882aecb40 x1874670563999488/t8589935248(8589935248) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826441 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1787826426.232245:0:55735:0:(file.c:7076:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000002:00010000:2.0:1787826426.232259:0:55735:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: layout flags 0 00010000:00010000:2.0:1787826426.232329:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.232341:0:55735:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.232368:0:55735:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55735 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826426.232392:0:55735:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8b40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670564003072:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826426.232502:0:55735:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35e40 going to sleep for 11 seconds 00000100:00100000:2.0:1787826426.236343:0:55735:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8b40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670564003072:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826426.236401:0:55735:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8389d expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.236420:0:55735:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8389d expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826426.236437:0:55735:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff976889ef8b40 x1874670564003072/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 376/584 e 0 to 0 dl 1787826442 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826426.236450:0:55735:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8389d expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787826426.236467:0:55735:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8389d expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.236482:0:55735:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:2.0:1787826426.236490:0:55735:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1787826426.236509:0:55735:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab17e5 00000080:00010000:2.0:1787826426.236519:0:55735:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab17e5 from it ffffb2d44221bbc8 00010000:00010000:2.0:1787826426.236524:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8389d expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.236544:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8389d expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.236584:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8389d expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.236596:0:55735:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8389d expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1787826426.236606:0:55735:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8389d expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.236652:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b8389d expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.236661:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8389d expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826426.236681:0:55735:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787826426.236698:0:55735:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787826426.236733:0:55735:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:2.0:1787826426.236747:0:55735:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8389d expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.236756:0:55735:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8389d expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1787826426.236763:0:55735:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b8389d expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826426.236780:0:55735:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787826426.236786:0:55735:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787826426.236794:0:55735:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000008:00010000:2.0:1787826426.236810:0:55735:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 0 [cma]time: 1787826425 1787826412 1787826412 by oinfo size 0 blocks 0 [cma]time 1787826426 1787826412 1787826412 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 1/0,0 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b83888 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826426.236940:0:55735:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 4294967294 -> 38 00010000:00010000:2.0:1787826426.236946:0:55735:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b8389d expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.236954:0:55735:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8389d expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:2.0:1787826426.237241:0:55735:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 1154002 00000002:00100000:2.0:1787826426.237302:0:55735:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff97688cb5da40 x1874670563997568/t8589935244(8589935244) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826441 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787826426.237316:0:55735:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787826426.237322:0:55735:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889ef8b40 x1874670564003328/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787826426.237348:0:55735:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8b40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670564003328:192.168.203.139@tcp:35:lfs.0 00000100:00100000:2.0:1787826426.237471:0:55735:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35a80 going to sleep for 11 seconds 00000100:00100000:2.0:1787826426.240589:0:55735:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8b40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670564003328:192.168.203.139@tcp:35:lfs.0 00000002:00100000:2.0:1787826426.240637:0:55735:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff97688cb5da40 x1874670563997568/t8589935244(8589935244) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826441 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1787826426.240728:0:55735:0:(file.c:479:ll_file_release()) START file . :VOLATILE:0000:4206CB4A:fd=03:[0x200000405:0x42:0x0](ffff9768901aba88), flags 554001 00000002:00100000:2.0:1787826426.240788:0:55735:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976882aec780 x1874670563997952/t8589935245(8589935245) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826441 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787826426.240799:0:55735:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787826426.240805:0:55735:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889ef8b40 x1874670564003584/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787826426.240871:0:55735:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8b40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670564003584:192.168.203.139@tcp:35:lfs.0 00000100:00100000:2.0:1787826426.240977:0:55735:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35a80 going to sleep for 11 seconds 00000100:00100000:2.0:1787826426.246076:0:55735:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8b40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55735:1874670564003584:192.168.203.139@tcp:35:lfs.0 00000002:00100000:2.0:1787826426.246124:0:55735:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976882aec780 x1874670563997952/t8589935245(8589935245) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826441 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826426.276270:0:55737:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826426.276311:0:55737:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826426.276336:0:55737:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826426.277780:0:55737:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826426.277797:0:55737:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826426.277810:0:55737:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826426.277815:0:55737:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826426.277919:0:55737:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44227bb58 00010000:00010000:0.0:1787826426.277926:0:55737:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826426.277937:0:55737:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826426.277959:0:55737:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826426.277969:0:55737:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=24 00000080:00200000:0.0:1787826426.277976:0:55737:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826426.277992:0:55737:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), flags=8 00000080:00200000:0.0:1787826426.277997:0:55737:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=getattr 00000002:00010000:0.0:1787826426.278021:0:55737:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f60b.sanity-flr,[0x0:0x0:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787826426.278091:0:55737:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55737 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.278100:0:55737:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55737 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.278111:0:55737:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55737 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826426.278149:0:55737:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55737:1874670564003712:192.168.203.139@tcp:101:getfattr.0 00000100:00100000:0.0:1787826426.278291:0:55737:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e900 going to sleep for 11 seconds 00000100:00100000:0.0:1787826426.283758:0:55737:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55737:1874670564003712:192.168.203.139@tcp:101:getfattr.0 00010000:00010000:0.0:1787826426.283791:0:55737:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 4/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.283819:0:55737:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.283856:0:55737:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.283870:0:55737:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.283883:0:55737:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.283898:0:55737:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.283907:0:55737:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.283915:0:55737:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826426.283934:0:55737:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff976882aee940 x1874670564003712/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 584/824 e 0 to 0 dl 1787826442 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'getfattr.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826426.283947:0:55737:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826426.283963:0:55737:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.283973:0:55737:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/29 (0 0) 00000080:00200000:0.0:1787826426.283981:0:55737:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826426.284038:0:55737:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x1d:0x0], updating i_size 0 00000080:00200000:0.0:1787826426.284042:0:55737:0:(namei.c:147:ll_iget()) got inode: [0x200000405:0x1d:0x0](ffff9768901a80c8): rc = 0 00000080:00200000:0.0:1787826426.284056:0:55737:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 38 -> 38 00000080:00010000:0.0:1787826426.284062:0:55737:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab17ec 00000080:00010000:0.0:1787826426.284074:0:55737:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:0.0:1787826426.284078:0:55737:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab17ec from it ffffb2d44227bb20 00010000:00010000:0.0:1787826426.284081:0:55737:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.284091:0:55737:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1787826426.284130:0:55737:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826426.284137:0:55737:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:0.0:1787826426.284164:0:55737:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826426.284175:0:55737:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826426.284196:0:55737:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826426.284204:0:55737:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826426.284209:0:55737:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:0.0:1787826426.284213:0:55737:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab17ec 00000080:00010000:0.0:1787826426.284216:0:55737:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab17ec from it ffffb2d44227bcb8 00010000:00010000:0.0:1787826426.284220:0:55737:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826426.284228:0:55737:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826426.284268:0:55737:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826426.284275:0:55737:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826426.284283:0:55737:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826426.284291:0:55737:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826426.284297:0:55737:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826426.284300:0:55737:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826426.284302:0:55737:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826426.284306:0:55737:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44227bbf8 00010000:00010000:0.0:1787826426.284309:0:55737:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826426.284318:0:55737:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826426.284325:0:55737:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826426.284332:0:55737:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:0.0:1787826426.284336:0:55737:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826426.284339:0:55737:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=9 00000080:00200000:0.0:1787826426.284345:0:55737:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:0.0:1787826426.284351:0:55737:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200000:0.0:1787826426.284436:0:55737:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826426.284444:0:55737:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826426.284450:0:55737:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826426.284459:0:55737:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826426.284466:0:55737:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826426.284470:0:55737:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826426.284472:0:55737:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826426.284476:0:55737:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44227bbf8 00010000:00010000:0.0:1787826426.284479:0:55737:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826426.284487:0:55737:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826426.284500:0:55737:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826426.284504:0:55737:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:0.0:1787826426.284507:0:55737:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826426.284510:0:55737:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=9 00000080:00200000:0.0:1787826426.284517:0:55737:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:0.0:1787826426.284521:0:55737:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200000:2.0:1787826426.323627:0:55741:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:2.0:1787826426.323676:0:55741:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826426.323702:0:55741:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826426.323720:0:55741:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826426.323729:0:55741:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826426.323739:0:55741:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:2.0:1787826426.323743:0:55741:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:2.0:1787826426.323752:0:55741:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d442283bf8 00010000:00010000:2.0:1787826426.323757:0:55741:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826426.323766:0:55741:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826426.323785:0:55741:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:2.0:1787826426.323794:0:55741:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=27 00000080:00200000:2.0:1787826426.323801:0:55741:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787826426.323806:0:55741:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=267 00000080:00200000:2.0:1787826426.323925:0:55741:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:2.0:1787826426.323935:0:55741:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826426.323942:0:55741:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826426.323951:0:55741:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826426.323957:0:55741:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826426.323961:0:55741:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:2.0:1787826426.323964:0:55741:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:2.0:1787826426.323968:0:55741:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d442283bf8 00010000:00010000:2.0:1787826426.323971:0:55741:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826426.323980:0:55741:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826426.323989:0:55741:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:2.0:1787826426.323994:0:55741:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=267 00000080:00200000:2.0:1787826426.323997:0:55741:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 44 00000080:00200008:2.0:1787826426.324006:0:55741:0:(file.c:1090:ll_file_open()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00000080:00200080:2.0:1787826426.324045:0:55741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8) cmd=c0086915 arg=17f73d0 00000100:00100000:2.0:1787826426.324136:0:55741:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55741:1874670564003840:192.168.203.139@tcp:34:lfs.0 00000100:00100000:2.0:1787826426.324286:0:55741:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35a80 going to sleep for 11 seconds 00000100:00100000:2.0:1787826426.331753:0:55741:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55741:1874670564003840:192.168.203.139@tcp:34:lfs.0 00010000:00010000:0.0:1787826426.335020:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:3.0:1787826426.339669:0:55741:0:(file.c:479:ll_file_release()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00010000:00010000:3.0:1787826426.339722:0:55741:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9600/0x8e7c99ee62ab0c1c lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b818c9 expref: -99 pid: 55483 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000080:00200000:3.0:1787826426.360780:0:55743:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826426.361103:0:55743:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826426.361130:0:55743:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826426.361150:0:55743:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826426.361162:0:55743:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826426.361172:0:55743:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826426.361185:0:55743:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826426.361195:0:55743:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44229bbf8 00010000:00010000:3.0:1787826426.361200:0:55743:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826426.361210:0:55743:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826426.361230:0:55743:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826426.361238:0:55743:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:3.0:1787826426.361246:0:55743:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787826426.361250:0:55743:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:3.0:1787826426.361255:0:55743:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 44 00000080:00200008:3.0:1787826426.361262:0:55743:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154400 00000002:00010000:3.0:1787826426.361283:0:55743:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114401 00010000:00010000:3.0:1787826426.361472:0:55743:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab17f3 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 55743 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.361481:0:55743:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab17f3 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55743 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787826426.361497:0:55743:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787826426.361512:0:55743:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff97688cb5c780 x1874670564004096/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787826426.361522:0:55743:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab17f3 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55743 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826426.361617:0:55743:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564004096:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826426.364299:0:55743:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25540 going to sleep for 11 seconds 00000100:00100000:3.0:1787826426.367768:0:55743:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564004096:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826426.367806:0:55743:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab17f3 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b838e3 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.367941:0:55743:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab17f3 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b838e3 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.367956:0:55743:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab17f3 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b838e3 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826426.367976:0:55743:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff97688cb5c780 x1874670564004096/t8589935257(8589935257) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826426.367991:0:55743:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff97688cb5c780 x1874670564004096/t8589935257(8589935257) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826426.368004:0:55743:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab17f3 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b838e3 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826426.368022:0:55743:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab17f3 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b838e3 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.368038:0:55743:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826426.368045:0:55743:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826426.368109:0:55743:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x1d:0x0], updating i_size 0 00000080:00200000:3.0:1787826426.368128:0:55743:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 38 -> 38 00000080:00010000:3.0:1787826426.368134:0:55743:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab17f3 00000080:00010000:3.0:1787826426.368142:0:55743:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab17f3 from it ffffb2d44229bbf0 00010000:00010000:3.0:1787826426.368147:0:55743:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab17f3 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b838e3 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.368159:0:55743:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab17f3 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b838e3 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000002:00100000:3.0:1787826426.368203:0:55743:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff97688cb5c780 x1874670564004096/t8589935257(8589935257) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1787826426.368658:0:55743:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:3.0:1787826426.368671:0:55743:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:3.0:1787826426.368706:0:55743:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826426.368713:0:55743:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:3.0:1787826426.368742:0:55743:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826426.368756:0:55743:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826426.368769:0:55743:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826426.368777:0:55743:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826426.368783:0:55743:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:3.0:1787826426.368787:0:55743:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab17ec 00000080:00010000:3.0:1787826426.368791:0:55743:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab17ec from it ffffb2d44229bcd8 00010000:00010000:3.0:1787826426.368794:0:55743:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826426.368802:0:55743:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:3.0:1787826426.368958:0:55743:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154000 00010000:00010000:3.0:1787826426.368973:0:55743:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab17f3 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b838e3 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826426.369131:0:55743:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826426.369143:0:55743:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826426.369154:0:55743:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826426.369163:0:55743:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826426.369170:0:55743:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826426.369174:0:55743:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826426.369187:0:55743:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826426.369192:0:55743:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44229bbf8 00010000:00010000:3.0:1787826426.369195:0:55743:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826426.369203:0:55743:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826426.369212:0:55743:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826426.369220:0:55743:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:3.0:1787826426.369225:0:55743:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787826426.369230:0:55743:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:3.0:1787826426.369234:0:55743:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 44 00000080:00200008:3.0:1787826426.369242:0:55743:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154400 00000080:00200000:3.0:1787826426.369284:0:55743:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:3.0:1787826426.369289:0:55743:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:3.0:1787826426.369308:0:55743:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826426.369313:0:55743:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:3.0:1787826426.369323:0:55743:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826426.369332:0:55743:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826426.369340:0:55743:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826426.369347:0:55743:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826426.369487:0:55743:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:3.0:1787826426.369490:0:55743:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab17ec 00000080:00010000:3.0:1787826426.369494:0:55743:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab17ec from it ffffb2d44229bcd8 00010000:00010000:3.0:1787826426.369498:0:55743:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826426.369506:0:55743:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200008:3.0:1787826426.369529:0:55743:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154000 00010000:00010000:3.0:1787826426.369743:0:55743:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab17f3 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b838e3 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826426.369792:0:55743:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826426.369803:0:55743:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826426.369810:0:55743:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826426.369819:0:55743:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826426.370234:0:55743:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826426.370241:0:55743:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826426.370244:0:55743:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826426.370250:0:55743:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44229bbf8 00010000:00010000:3.0:1787826426.370254:0:55743:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826426.370263:0:55743:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826426.370277:0:55743:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826426.370286:0:55743:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:2.0:1787826426.370587:0:55743:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787826426.370595:0:55743:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:2.0:1787826426.370600:0:55743:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 46 00000080:00200008:2.0:1787826426.370609:0:55743:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154402 00000002:00010000:2.0:1787826426.370634:0:55743:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114403 00010000:00010000:2.0:1787826426.370704:0:55743:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab17fa lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 55743 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.370713:0:55743:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab17fa lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55743 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1787826426.370734:0:55743:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1787826426.370750:0:55743:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efa940 x1874670564004352/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1787826426.370761:0:55743:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab17fa lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55743 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826426.370789:0:55743:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564004352:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826426.371082:0:55743:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35a80 going to sleep for 11 seconds 00000100:00100000:1.0:1787826426.374890:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826426.374993:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585292672 00000100:00100000:0.0:1787826426.375007:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585292672 00000100:00100000:0.0:1787826426.375079:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 610 00000100:00100000:0.0:1787826426.375091:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0161a40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:16564:x1874670585292672:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826426.375114:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826426.375223:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826426.375230:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0161a40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:16564:x1874670585292672:12345-192.168.203.139@tcp:104: Request processed in 141us (356us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826426.375238:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 610 00010000:00010000:3.0:1787826426.375401:0:49796:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826426.375410:0:49796:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826426.375420:0:49796:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925c600 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826426.375426:0:49796:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826426.375435:0:49796:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826426.375443:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826426.375477:0:49796:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab17f3 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b838e3 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826426.375532:0:49796:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 2/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826426.375540:0:49796:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826426.375583:0:49796:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 1/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826426.375590:0:49796:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab17ec lrc: 0/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b838c0 expref: -99 pid: 55737 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826426.375607:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826426.375609:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826426.375610:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826426.375618:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787826426.375636:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564004480:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826426.379796:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564004480:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826426.379916:0:55743:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564004352:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826426.379939:0:55743:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode CW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab17fa lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b838f8 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.379951:0:55743:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab17fa lrc: 4/1,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b838f8 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.379962:0:55743:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab17fa lrc: 4/1,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b838f8 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.379972:0:55743:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab17fa lrc: 4/1,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b838f8 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.379982:0:55743:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab17fa lrc: 5/1,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b838f8 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.379989:0:55743:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab17fa lrc: 5/1,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b838f8 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.379999:0:55743:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab17fa lrc: 4/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b838f8 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826426.380011:0:55743:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff976889efa940 x1874670564004352/t8589935258(8589935258) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787826426.380025:0:55743:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efa940 x1874670564004352/t8589935258(8589935258) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826426.380043:0:55743:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab17fa lrc: 3/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b838f8 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.380059:0:55743:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 8 res 8589935621/29 (0 0) 00000080:00200000:2.0:1787826426.380067:0:55743:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787826426.380119:0:55743:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x1d:0x0], updating i_size 0 00000080:00010000:2.0:1787826426.380123:0:55743:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab17fa 00000080:00010000:2.0:1787826426.380129:0:55743:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab17fa from it ffffb2d44229bbf0 00010000:00010000:2.0:1787826426.380132:0:55743:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab17fa lrc: 3/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b838f8 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.380138:0:55743:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab17fa lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b838f8 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:2.0:1787826426.380169:0:55743:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efa940 x1874670564004352/t8589935258(8589935258) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:2.0:1787826426.380347:0:55743:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=800466af arg=7ffceeac907c 00000080:00200080:2.0:1787826426.380474:0:55743:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=80086601 arg=7ffceeac9018 00000100:00100000:2.0:1787826426.380527:0:55743:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564004608:192.168.203.139@tcp:33:lfs.0 00000100:00100000:2.0:1787826426.380738:0:55743:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35a80 going to sleep for 11 seconds 00000100:00100000:2.0:1787826426.382738:0:55743:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564004608:192.168.203.139@tcp:33:lfs.0 00000080:00200080:2.0:1787826426.382788:0:55743:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=7ffceeac9004 00000002:00010000:2.0:1787826426.382868:0:55743:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 01004040000002 00010000:00010000:2.0:1787826426.382893:0:55743:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab17fa lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x22153cd396b838f8 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.382934:0:55743:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab17f3 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b838e3 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826426.382951:0:55743:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b8389d expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000002:00100000:2.0:1787826426.383002:0:55743:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff97688cb5c780 x1874670564004096/t8589935257(8589935257) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826442 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787826426.383015:0:55743:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787826426.383021:0:55743:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efb0c0 x1874670564004864/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787826426.383046:0:55743:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564004864:192.168.203.139@tcp:35:lfs.0 00000100:00100000:2.0:1787826426.383129:0:55743:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35a80 going to sleep for 11 seconds 00000100:00100000:2.0:1787826426.385456:0:55743:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564004864:192.168.203.139@tcp:35:lfs.0 00000002:00100000:2.0:1787826426.385627:0:55743:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff97688cb5c780 x1874670564004096/t8589935257(8589935257) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826442 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1787826426.385735:0:55743:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab17fa lrc: 1/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b838f8 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.385748:0:55743:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab17f3 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b838e3 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826426.385754:0:55743:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787826426.385758:0:55743:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab17fa lrc: 0/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b838f8 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.385772:0:55743:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9a00/0x8e7c99ee62ab17f3 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b838e3 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826426.385802:0:55743:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab1801 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 55743 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.385810:0:55743:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x81000000001000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab1801 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 55743 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1787826426.385821:0:55743:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1787826426.385885:0:55743:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efb0c0 x1874670564005120/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 976/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1787826426.385896:0:55743:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab1801 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x81000000000000 pid: 55743 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826426.385920:0:55743:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564005120:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826426.386707:0:55743:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35a80 going to sleep for 11 seconds 00000100:00100000:2.0:1787826426.390228:0:55743:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564005120:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826426.390265:0:55743:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab1801 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b8390d expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.390281:0:55743:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab1801 lrc: 4/1,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b8390d expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.390293:0:55743:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab1801 lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b8390d expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.390301:0:55743:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab1801 lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b8390d expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.390313:0:55743:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab1801 lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b8390d expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.390320:0:55743:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab1801 lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b8390d expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.390324:0:55743:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab1801 lrc: 4/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b8390d expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826426.390334:0:55743:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x600002b, status=0 req@ffff976889efb0c0 x1874670564005120/t8589935260(8589935260) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 976/848 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787826426.390344:0:55743:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efb0c0 x1874670564005120/t8589935260(8589935260) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 976/848 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826426.390434:0:55743:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab1801 lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b8390d expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.390452:0:55743:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 2 res 8589935621/29 (0 0) 00000080:00200000:2.0:1787826426.390458:0:55743:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000002:00100000:2.0:1787826426.390480:0:55743:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efb0c0 x1874670564005120/t8589935260(8589935260) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 768/848 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1787826426.390487:0:55743:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1801 00000080:00010000:2.0:1787826426.390494:0:55743:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1801 from it ffffb2d44229bcc0 00010000:00010000:2.0:1787826426.390497:0:55743:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab1801 lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b8390d expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.390504:0:55743:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab1801 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b8390d expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200080:2.0:1787826426.390669:0:55743:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=e652e0 00010000:00010000:2.0:1787826426.390727:0:55743:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab1801 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0x22153cd396b8390d expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.390743:0:55743:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883515800/0x8e7c99ee62ab1801 lrc: 0/0,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x85809400000000 nid: local remote: 0x22153cd396b8390d expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826426.390762:0:55743:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889efb0c0 x1874670564005120/t8589935260(8589935260) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 768/848 e 0 to 0 dl 1787826442 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787826426.390773:0:55743:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787826426.390780:0:55743:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efb840 x1874670564005376/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 504/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787826426.390805:0:55743:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564005376:192.168.203.139@tcp:35:lfs.0 00000100:00100000:2.0:1787826426.390956:0:55743:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35a80 going to sleep for 11 seconds 00000100:00100000:0.0:1787826426.392996:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826426.393810:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585292800 00000100:00100000:1.0:1787826426.393920:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585292800 00000100:00100000:1.0:1787826426.394048:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 611 00000100:00100000:1.0:1787826426.394061:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aee940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6521:x1874670585292800:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826426.394084:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b8389d expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787826426.394196:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826426.394204:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aee940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6521:x1874670585292800:12345-192.168.203.139@tcp:104: Request processed in 146us (1220us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826426.394211:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 611 00010000:00010000:2.0:1787826426.394274:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826426.394283:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b8389d expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826426.394293:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514800 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:2.0:1787826426.394298:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b8389d expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826426.394307:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b8389d expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826426.394315:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b8389d expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1787826426.394348:0:42220:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x1d:0x0]: losing layout lock 00010000:00010000:2.0:1787826426.394399:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b8389d expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826426.394407:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826426.394440:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b8389d expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826426.394449:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab17e5 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b8389d expref: -99 pid: 55735 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826426.394476:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.394477:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.394479:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826426.394489:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1787826426.394508:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564005504:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826426.402882:0:55743:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564005376:192.168.203.139@tcp:35:lfs.0 00000080:00020000:1.0:1787826426.402907:0:55743:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9768877a4800: inode [0x200000405:0x1d:0x0] mdc close failed: rc = -22 00000002:00100000:1.0:1787826426.402931:0:55743:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889efb0c0 x1874670564005120/t8589935260(8589935260) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 768/848 e 0 to 0 dl 1787826442 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1787826426.402974:0:55743:0:(file.c:7076:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000002:00010000:1.0:1787826426.402987:0:55743:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: layout flags 0 00010000:00010000:1.0:1787826426.403053:0:55743:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf600/0x8e7c99ee62ab1808 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55743 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.403064:0:55743:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf600/0x8e7c99ee62ab1808 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55743 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.403078:0:55743:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf600/0x8e7c99ee62ab1808 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55743 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826426.403102:0:55743:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b01603c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564005632:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826426.403208:0:55743:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3d80 going to sleep for 11 seconds 00000100:00100000:2.0:1787826426.404241:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564005504:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826426.407979:0:55743:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b01603c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564005632:192.168.203.139@tcp:101:lfs.0 00010000:00010000:1.0:1787826426.408157:0:55743:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf600/0x8e7c99ee62ab1808 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83922 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.408184:0:55743:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf600/0x8e7c99ee62ab1808 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83922 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787826426.408203:0:55743:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9768b01603c0 x1874670564005632/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 376/584 e 0 to 0 dl 1787826442 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787826426.408216:0:55743:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf600/0x8e7c99ee62ab1808 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83922 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787826426.408234:0:55743:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf600/0x8e7c99ee62ab1808 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83922 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.408251:0:55743:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:1.0:1787826426.408260:0:55743:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1787826426.408282:0:55743:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1808 00000080:00010000:1.0:1787826426.408292:0:55743:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1808 from it ffffb2d44229bbc8 00010000:00010000:1.0:1787826426.408297:0:55743:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf600/0x8e7c99ee62ab1808 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83922 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.408314:0:55743:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf600/0x8e7c99ee62ab1808 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83922 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826426.408345:0:55743:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf600/0x8e7c99ee62ab1808 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83922 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826426.408481:0:55743:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf600/0x8e7c99ee62ab1808 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83922 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1787826426.408491:0:55743:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf600/0x8e7c99ee62ab1808 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83922 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787826426.408649:0:55743:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 4294967294 -> 38 00010000:00010000:1.0:1787826426.408657:0:55743:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf600/0x8e7c99ee62ab1808 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83922 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826426.408666:0:55743:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf600/0x8e7c99ee62ab1808 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83922 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787826426.409300:0:55743:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826426.409326:0:55743:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826426.409338:0:55743:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826426.409363:0:55743:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826426.409377:0:55743:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826426.409384:0:55743:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826426.409388:0:55743:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826426.409393:0:55743:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44229bbf8 00010000:00010000:1.0:1787826426.409396:0:55743:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826426.409404:0:55743:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826426.409415:0:55743:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826426.409424:0:55743:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=24 00000080:00200000:1.0:1787826426.409431:0:55743:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826426.409449:0:55743:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787826426.409455:0:55743:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=. :VOLATILE:0000:741ECEE8:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), file ffff97688922b080, open_flags 2f9c2, mode 8180 opened 0 00000080:00200000:1.0:1787826426.409462:0:55743:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=. :VOLATILE:0000:741ECEE8:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=open|creat 00000002:00010000:1.0:1787826426.409488:0:55743:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: . :VOLATILE:0000:741ECEE8:fd=03,[0x200000405:0x43:0x0]) in obj [0x200000bd0:0x4:0x0], intent: open|creat flags 0574703 00010000:00010000:1.0:1787826426.409567:0:55743:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab180f lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55743 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.409575:0:55743:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab180f lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55743 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1787826426.409589:0:55743:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1787826426.409600:0:55743:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b01603c0 x1874670564005888/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/66640 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1787826426.409614:0:55743:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab180f lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55743 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826426.409640:0:55743:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b01603c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564005888:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826426.409867:0:55743:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3d80 going to sleep for 11 seconds 00000100:00100000:1.0:1787826426.413697:0:55743:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b01603c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564005888:192.168.203.139@tcp:101:lfs.0 00010000:00010000:1.0:1787826426.413734:0:55743:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab180f lrc: 4/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83930 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.413771:0:55743:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab180f lrc: 4/0,1 mode: --/PR res: [0x200000405:0x43:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83930 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.413780:0:55743:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab180f lrc: 4/0,1 mode: --/PR res: [0x200000405:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83930 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.413792:0:55743:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab180f lrc: 4/0,1 mode: PR/PR res: [0x200000405:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83930 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.413805:0:55743:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab180f lrc: 4/0,1 mode: PR/PR res: [0x200000405:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83930 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.413819:0:55743:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab180f lrc: 5/1,1 mode: PR/PR res: [0x200000405:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83930 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.413951:0:55743:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab180f lrc: 5/1,1 mode: PR/PR res: [0x200000405:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83930 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.413963:0:55743:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab180f lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83930 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787826426.413978:0:55743:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff9768b01603c0 x1874670564005888/t8589935262(8589935262) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1787826426.413992:0:55743:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9768b01603c0 x1874670564005888/t8589935262(8589935262) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787826426.414007:0:55743:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab180f lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83930 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.414019:0:55743:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/67 (0 0) 00000080:00200000:1.0:1787826426.414027:0:55743:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd0:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787826426.414046:0:55743:0:(namei.c:1628:ll_create_it()) VFS Op:name=. :VOLATILE:0000:741ECEE8:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=open|creat 00000080:00200000:1.0:1787826426.414089:0:55743:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000405:0x43:0x0](ffff9768901b3248) 00000080:00200000:1.0:1787826426.414099:0:55743:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x43:0x0], updating i_size 0 00000080:00200000:1.0:1787826426.414141:0:55743:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x43:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:1.0:1787826426.414146:0:55743:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000405:0x43:0x0](ffff9768901b3248) 00000080:00010000:1.0:1787826426.414149:0:55743:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x43:0x0](ffff9768901b3248) for lock 0x8e7c99ee62ab180f 00000080:00010000:1.0:1787826426.414163:0:55743:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x43:0x0](ffff9768901b3248) 00000080:00200008:1.0:1787826426.414193:0:55743:0:(file.c:1090:ll_file_open()) START file . :VOLATILE:0000:741ECEE8:fd=03:[0x200000405:0x43:0x0](ffff9768901b3248), flags 574702 00000002:00100000:1.0:1787826426.414221:0:55743:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9768b01603c0 x1874670564005888/t8589935262(8589935262) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1787826426.414262:0:55743:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab180f from it ffff976885d9ce40 00010000:00010000:1.0:1787826426.414267:0:55743:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab180f lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83930 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.414276:0:55743:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab180f lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83930 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787826426.414303:0:55743:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x43:0x0](ffff9768901b3248), inode mode 8180 mask 40 00000080:00200080:1.0:1787826426.414333:0:55743:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=7ffceeac9004 00000002:00010000:1.0:1787826426.414369:0:55743:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 01004040000002 00010000:00010000:1.0:1787826426.414431:0:55743:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde400/0x8e7c99ee62ab1816 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 55743 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.414440:0:55743:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x81000000001000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde400/0x8e7c99ee62ab1816 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 55743 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1787826426.414453:0:55743:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1787826426.414460:0:55743:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b0162d00 x1874670564006144/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1787826426.414469:0:55743:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde400/0x8e7c99ee62ab1816 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x81000000000000 pid: 55743 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826426.414492:0:55743:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0162d00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564006144:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826426.414924:0:55743:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3d80 going to sleep for 11 seconds 00000100:00100000:1.0:1787826426.419120:0:55743:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0162d00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564006144:192.168.203.139@tcp:101:lfs.0 00010000:00010000:1.0:1787826426.419153:0:55743:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde400/0x8e7c99ee62ab1816 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83945 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.419167:0:55743:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde400/0x8e7c99ee62ab1816 lrc: 4/1,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83945 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.419195:0:55743:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde400/0x8e7c99ee62ab1816 lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83945 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.419208:0:55743:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde400/0x8e7c99ee62ab1816 lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83945 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.419224:0:55743:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde400/0x8e7c99ee62ab1816 lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83945 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.419232:0:55743:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde400/0x8e7c99ee62ab1816 lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83945 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.419240:0:55743:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde400/0x8e7c99ee62ab1816 lrc: 4/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83945 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787826426.419251:0:55743:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x600002b, status=0 req@ffff9768b0162d00 x1874670564006144/t8589935263(8589935263) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1787826426.419265:0:55743:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9768b0162d00 x1874670564006144/t8589935263(8589935263) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787826426.419282:0:55743:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde400/0x8e7c99ee62ab1816 lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83945 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.419297:0:55743:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 2 res 8589935621/29 (0 0) 00000080:00200000:1.0:1787826426.419304:0:55743:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000002:00100000:1.0:1787826426.419327:0:55743:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9768b0162d00 x1874670564006144/t8589935263(8589935263) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1787826426.419336:0:55743:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1816 00000080:00010000:1.0:1787826426.419343:0:55743:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1816 from it ffffb2d44229bcc0 00010000:00010000:1.0:1787826426.419347:0:55743:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde400/0x8e7c99ee62ab1816 lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83945 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.419460:0:55743:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde400/0x8e7c99ee62ab1816 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83945 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200080:1.0:1787826426.419691:0:55743:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=e65300 00010000:00010000:1.0:1787826426.419747:0:55743:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde400/0x8e7c99ee62ab1816 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0x22153cd396b83945 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826426.419763:0:55743:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fde400/0x8e7c99ee62ab1816 lrc: 0/0,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x85809400000000 nid: local remote: 0x22153cd396b83945 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787826426.419781:0:55743:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9768b0162d00 x1874670564006144/t8589935263(8589935263) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826442 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1787826426.419792:0:55743:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1787826426.419798:0:55743:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b0162580 x1874670564006400/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 504/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1787826426.419821:0:55743:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0162580 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564006400:192.168.203.139@tcp:35:lfs.0 00000100:00100000:1.0:1787826426.419989:0:55743:0:(client.c:2715:ptlrpc_set_wait()) set ffff976888ed3d80 going to sleep for 11 seconds 00000100:00100000:0.0:1787826426.423145:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826426.423255:0:42211:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1787826426.423279:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585292928 00000100:00100000:1.0:1787826426.423290:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585292928 00000100:00100000:1.0:1787826426.423505:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 612 00000100:00100000:1.0:1787826426.423518:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768895bb050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585292928:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826426.423542:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf600/0x8e7c99ee62ab1808 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83922 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826426.423755:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826426.423763:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768895bb050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585292928:12345-192.168.203.139@tcp:104: Request processed in 249us (632us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826426.423772:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 612 00010000:00010000:2.0:1787826426.423855:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826426.423865:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf600/0x8e7c99ee62ab1808 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83922 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.423875:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976890fdf600 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:2.0:1787826426.423881:0:42221:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf600/0x8e7c99ee62ab1808 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83922 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.423891:0:42221:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf600/0x8e7c99ee62ab1808 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83922 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.423901:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf600/0x8e7c99ee62ab1808 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b83922 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826426.423932:0:42221:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x1d:0x0]: losing layout lock 00010000:00010000:2.0:1787826426.423984:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf600/0x8e7c99ee62ab1808 lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83922 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.423992:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1787826426.424066:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564006528:192.168.203.139@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1787826426.424167:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf600/0x8e7c99ee62ab1808 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83922 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.424183:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf600/0x8e7c99ee62ab1808 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83922 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.424211:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.424213:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.424215:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826426.428558:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826426.428735:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585293056 00000100:00100000:1.0:1787826426.428747:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585293056 00000100:00100000:1.0:1787826426.428809:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 613 00000100:00100000:1.0:1787826426.428819:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aef480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585293056:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826426.428870:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab180f lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83930 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826426.428956:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826426.428962:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aef480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585293056:12345-192.168.203.139@tcp:104: Request processed in 144us (434us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826426.428969:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 613 00000100:00100000:2.0:1787826426.429511:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564006528:192.168.203.139@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1787826426.429800:0:49796:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826426.429809:0:49796:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab180f lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83930 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826426.429821:0:49796:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976890fdee00 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826426.429971:0:49796:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab180f lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x43:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83930 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826426.429981:0:49796:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab180f lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x43:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83930 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826426.429990:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab180f lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x43:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b83930 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826426.430019:0:49796:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x43:0x0]: losing layout lock 00010000:00010000:1.0:1787826426.431708:0:49796:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab180f lrc: 2/0,0 mode: --/PR res: [0x200000405:0x43:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83930 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826426.431726:0:49796:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1787826426.431819:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670564006656:192.168.203.139@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1787826426.432065:0:49796:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab180f lrc: 1/0,0 mode: --/PR res: [0x200000405:0x43:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83930 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826426.432075:0:49796:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdee00/0x8e7c99ee62ab180f lrc: 0/0,0 mode: --/PR res: [0x200000405:0x43:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83930 expref: -99 pid: 55743 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826426.432103:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787826426.432106:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787826426.432109:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826426.440927:0:55743:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0162580 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564006400:192.168.203.139@tcp:35:lfs.0 00000002:00100000:2.0:1787826426.440978:0:55743:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9768b0162d00 x1874670564006144/t8589935263(8589935263) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826442 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1787826426.441035:0:55743:0:(file.c:7076:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000002:00010000:2.0:1787826426.441049:0:55743:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: layout flags 0 00010000:00010000:2.0:1787826426.441115:0:55743:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55743 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.441126:0:55743:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55743 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.441137:0:55743:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55743 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826426.441161:0:55743:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564006784:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826426.441276:0:55743:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35e40 going to sleep for 11 seconds 00000100:00100000:1.0:1787826426.444774:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0160b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670564006656:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826426.445095:0:55743:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564006784:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826426.445126:0:55743:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83961 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.445143:0:55743:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83961 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826426.445162:0:55743:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff976889ef9a40 x1874670564006784/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 376/480 e 0 to 0 dl 1787826442 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826426.445182:0:55743:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83961 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787826426.445199:0:55743:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83961 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.445216:0:55743:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:2.0:1787826426.445223:0:55743:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1787826426.445239:0:55743:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab181d 00000080:00010000:2.0:1787826426.445248:0:55743:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab181d from it ffffb2d44229bbc8 00010000:00010000:2.0:1787826426.445253:0:55743:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83961 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.445263:0:55743:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83961 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.445297:0:55743:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83961 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.445311:0:55743:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83961 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1787826426.445321:0:55743:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83961 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.445386:0:55743:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83961 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.445395:0:55743:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83961 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826426.445413:0:55743:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787826426.445426:0:55743:0:(lov_io.c:693:lov_io_mirror_init()) [0x200000405:0x1d:0x0]: flr state: 1, move mirror from 1 to 1, have retried: 0, mirror count: 2 00020000:00200000:2.0:1787826426.445429:0:55743:0:(lov_io.c:750:lov_io_mirror_init()) use delayed RPC state for this IO 00020000:00200000:2.0:1787826426.445438:0:55743:0:(lov_io.c:1253:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:2.0:1787826426.445474:0:55743:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:2.0:1787826426.445489:0:55743:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83961 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.445497:0:55743:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83961 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1787826426.445504:0:55743:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83961 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826426.445520:0:55743:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787826426.445525:0:55743:0:(lov_io.c:693:lov_io_mirror_init()) [0x200000405:0x1d:0x0]: flr state: 1, move mirror from 1 to 1, have retried: 0, mirror count: 2 00020000:00200000:2.0:1787826426.445527:0:55743:0:(lov_io.c:750:lov_io_mirror_init()) use delayed RPC state for this IO 00020000:00200000:2.0:1787826426.445529:0:55743:0:(lov_io.c:1253:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:2.0:1787826426.445763:0:55743:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787826426.445883:0:55743:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 4294967294 -> 39 00010000:00010000:2.0:1787826426.445890:0:55743:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83961 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.445897:0:55743:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83961 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:2.0:1787826426.445940:0:55743:0:(file.c:479:ll_file_release()) START file . :VOLATILE:0000:741ECEE8:fd=03:[0x200000405:0x43:0x0](ffff9768901b3248), flags 554002 00000002:00100000:2.0:1787826426.445995:0:55743:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9768b01603c0 x1874670564005888/t8589935262(8589935262) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826442 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787826426.446008:0:55743:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787826426.446014:0:55743:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889ef9a40 x1874670564007040/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787826426.446039:0:55743:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564007040:192.168.203.139@tcp:35:lfs.0 00000100:00100000:2.0:1787826426.446150:0:55743:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35c00 going to sleep for 11 seconds 00000100:00100000:2.0:1787826426.450165:0:55743:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef9a40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564007040:192.168.203.139@tcp:35:lfs.0 00000002:00100000:2.0:1787826426.450221:0:55743:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9768b01603c0 x1874670564005888/t8589935262(8589935262) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826442 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1787826426.450297:0:55743:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154002 00000002:00100000:2.0:1787826426.450384:0:55743:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889efa940 x1874670564004352/t8589935258(8589935258) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826442 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787826426.450396:0:55743:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787826426.450401:0:55743:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efbc00 x1874670564007296/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787826426.450428:0:55743:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564007296:192.168.203.139@tcp:35:lfs.0 00000100:00100000:2.0:1787826426.450534:0:55743:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35c00 going to sleep for 11 seconds 00000100:00100000:2.0:1787826426.454057:0:55743:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55743:1874670564007296:192.168.203.139@tcp:35:lfs.0 00000002:00100000:2.0:1787826426.454106:0:55743:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889efa940 x1874670564004352/t8589935258(8589935258) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826442 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826426.477664:0:55745:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826426.477711:0:55745:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826426.477737:0:55745:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826426.477757:0:55745:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826426.477766:0:55745:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826426.477775:0:55745:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826426.477779:0:55745:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826426.477789:0:55745:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44226bb58 00010000:00010000:0.0:1787826426.477794:0:55745:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826426.477804:0:55745:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826426.477852:0:55745:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826426.477861:0:55745:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=24 00000080:00200000:0.0:1787826426.477868:0:55745:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826426.477883:0:55745:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), flags=8 00000080:00200000:0.0:1787826426.477888:0:55745:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=getattr 00000002:00010000:0.0:1787826426.477906:0:55745:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f60b.sanity-flr,[0x0:0x0:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787826426.477971:0:55745:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55745 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.477979:0:55745:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55745 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.477990:0:55745:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55745 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826426.478028:0:55745:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55745:1874670564007424:192.168.203.139@tcp:101:getfattr.0 00000100:00100000:0.0:1787826426.478165:0:55745:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e9c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787826426.485880:0:55745:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55745:1874670564007424:192.168.203.139@tcp:101:getfattr.0 00010000:00010000:2.0:1787826426.485918:0:55745:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 4/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.485948:0:55745:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.485958:0:55745:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.485971:0:55745:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.485983:0:55745:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.485996:0:55745:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.486006:0:55745:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.486020:0:55745:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826426.486033:0:55745:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff976882aef840 x1874670564007424/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 584/720 e 0 to 0 dl 1787826442 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'getfattr.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826426.486046:0:55745:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787826426.486062:0:55745:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.486073:0:55745:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/29 (0 0) 00000080:00200000:2.0:1787826426.486082:0:55745:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787826426.486149:0:55745:0:(namei.c:147:ll_iget()) got inode: [0x200000405:0x1d:0x0](ffff9768901a80c8): rc = 0 00000080:00200000:2.0:1787826426.486166:0:55745:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 39 -> 39 00000080:00010000:2.0:1787826426.486172:0:55745:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1824 00000080:00010000:2.0:1787826426.486199:0:55745:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:2.0:1787826426.486203:0:55745:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1824 from it ffffb2d44226bb20 00010000:00010000:2.0:1787826426.486207:0:55745:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.486216:0:55745:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1787826426.486270:0:55745:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787826426.486278:0:55745:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:2.0:1787826426.486313:0:55745:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826426.486322:0:55745:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.486334:0:55745:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826426.486342:0:55745:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826426.486491:0:55745:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:2.0:1787826426.486496:0:55745:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1824 00000080:00010000:2.0:1787826426.486501:0:55745:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1824 from it ffffb2d44226bcb8 00010000:00010000:2.0:1787826426.486505:0:55745:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.486513:0:55745:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826426.486653:0:55745:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 37 restore needed 0 00000001:00010000:2.0:1787826426.486669:0:55745:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:2.0:1787826426.486695:0:55745:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787826426.486699:0:55745:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff97688757c430, osc lock ffff976883f7bc08, flags 0x400001002 00010000:00010000:2.0:1787826426.486721:0:55745:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.486734:0:55745:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.486745:0:55745:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.486754:0:55745:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.486802:0:55745:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab182b lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55745 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.486812:0:55745:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab182b lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55745 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1787826426.486904:0:55745:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:2.0:1787826426.486923:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1787826426.486945:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564007552:192.168.203.139@tcp:101:getfattr.0 00000100:00100000:0.0:1787826426.492127:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826426.492189:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585293312 00000100:00100000:0.0:1787826426.492200:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585293312 00000100:00100000:0.0:1787826426.492268:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 614 00000100:00100000:0.0:1787826426.492280:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aec000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:8435:x1874670585293312:12345-192.168.203.139@tcp:106: 00010000:00010000:0.0:1787826426.492300:0:42211:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826426.492315:0:42211:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:0.0:1787826426.492349:0:42211:0:(cl_object.c:322:cl_object_glimpse()) header@00000000c0eb4eb5[0x110, 1, [0x200000405:0x42:0x0] hash] 00000020:00010000:0.0:1787826426.492366:0:42211:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826426 blocks: 0 00000020:00010000:0.0:1787826426.492370:0:42211:0:(cl_object.c:322:cl_object_glimpse()) header@0000000048e089aa[0x116, 2, [0x280000401:0x10e:0x0]] 00000020:00010000:0.0:1787826426.492372:0:42211:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826426 blocks: 0 00000100:00100000:0.0:1787826426.492445:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aec000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:8435:x1874670585293312:12345-192.168.203.139@tcp:106: Request processed in 167us (334us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826426.492453:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 614 00010000:00010000:3.0:1787826426.497386:0:2403:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab182b lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55745 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.497415:0:2403:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab182b lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55745 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.497433:0:2403:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab182b lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55745 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.497442:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab182b lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55745 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.497457:0:2403:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab182b lrc: 5/1,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55745 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826426.497473:0:2403:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:3.0:1787826426.497482:0:2403:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826426.497500:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab182b lrc: 4/1,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55745 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.497509:0:2403:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab182b lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55745 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.497518:0:2403:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab182b lrc: 4/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55745 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.497527:0:2403:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515800 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826426.497533:0:2403:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab182b lrc: 5/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55745 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.497660:0:2403:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826426.497670:0:2403:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab182b lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55745 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.497679:0:2403:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c07e43af/0x8e7c99ee62ab182b lrc: 0/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55745 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826426.497709:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564007552:192.168.203.139@tcp:101:getfattr.0 00000080:00200000:0.0:1787826426.497970:0:55745:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787826426.497978:0:55745:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9768ad1fd8b0 (ffff976890d10098/0) at cl_lock_release():245 00000020:00010000:0.0:1787826426.497982:0:55745:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768ad1fd8b0 (ffff976890d10098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826426.497988:0:55745:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757c430 (ffff976890d103f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826426.497992:0:55745:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768ad1fd8b0 (ffff976890d10098/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826426.497996:0:55745:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757c430 (ffff976890d10098/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826426.498012:0:55745:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 39 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826426.498065:0:55745:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826426.498091:0:55745:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826426.498111:0:55745:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826426.498126:0:55745:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826426.498134:0:55745:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826426.498142:0:55745:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826426.498147:0:55745:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826426.498153:0:55745:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44226bbf8 00010000:00010000:0.0:1787826426.498157:0:55745:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826426.498165:0:55745:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826426.498191:0:55745:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826426.498199:0:55745:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:0.0:1787826426.498205:0:55745:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826426.498210:0:55745:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=9 00000080:00200000:0.0:1787826426.498219:0:55745:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:0.0:1787826426.498227:0:55745:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200000:0.0:1787826426.498286:0:55745:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826426.498294:0:55745:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826426.498302:0:55745:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826426.498311:0:55745:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826426.498318:0:55745:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826426.498322:0:55745:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826426.498325:0:55745:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826426.498329:0:55745:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d44226bbf8 00010000:00010000:0.0:1787826426.498332:0:55745:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826426.498340:0:55745:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826426.498348:0:55745:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826426.498352:0:55745:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:0.0:1787826426.498367:0:55745:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826426.498371:0:55745:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=9 00000080:00200000:0.0:1787826426.498379:0:55745:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:0.0:1787826426.498383:0:55745:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200000:0.0:1787826426.537879:0:55749:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826426.537918:0:55749:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826426.537939:0:55749:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826426.537958:0:55749:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826426.537968:0:55749:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826426.537976:0:55749:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826426.537980:0:55749:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826426.537989:0:55749:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d442283bf8 00010000:00010000:0.0:1787826426.537993:0:55749:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826426.538003:0:55749:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826426.538022:0:55749:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826426.538031:0:55749:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=27 00000080:00200000:0.0:1787826426.538038:0:55749:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826426.538042:0:55749:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=267 00000080:00200000:0.0:1787826426.538074:0:55749:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826426.538081:0:55749:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826426.538088:0:55749:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826426.538097:0:55749:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826426.538103:0:55749:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826426.538107:0:55749:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826426.538110:0:55749:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826426.538114:0:55749:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d442283bf8 00010000:00010000:0.0:1787826426.538117:0:55749:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826426.538125:0:55749:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826426.538133:0:55749:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826426.538137:0:55749:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=267 00000080:00200000:0.0:1787826426.538141:0:55749:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 44 00000080:00200008:0.0:1787826426.538147:0:55749:0:(file.c:1090:ll_file_open()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00000080:00200080:0.0:1787826426.538326:0:55749:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8) cmd=c0086915 arg=d133d0 00000100:00100000:0.0:1787826426.538524:0:55749:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55749:1874670564007680:192.168.203.139@tcp:34:lfs.0 00000100:00100000:0.0:1787826426.538723:0:55749:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741ed80 going to sleep for 11 seconds 00000100:00100000:1.0:1787826426.542163:0:55749:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55749:1874670564007680:192.168.203.139@tcp:34:lfs.0 00000080:00200008:1.0:1787826426.543339:0:55749:0:(file.c:479:ll_file_release()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00010000:00010000:1.0:1787826426.543504:0:55749:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9600/0x8e7c99ee62ab0c1c lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b818c9 expref: -99 pid: 55483 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000080:00200000:3.0:1787826426.562481:0:55751:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826426.562525:0:55751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826426.562549:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826426.562569:0:55751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826426.562579:0:55751:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826426.562587:0:55751:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826426.562591:0:55751:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826426.562601:0:55751:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4422abbf8 00010000:00010000:3.0:1787826426.562605:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826426.562616:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826426.562633:0:55751:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826426.562642:0:55751:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:3.0:1787826426.562648:0:55751:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787826426.562652:0:55751:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:3.0:1787826426.562656:0:55751:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 44 00000080:00200008:3.0:1787826426.562662:0:55751:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154400 00000002:00010000:3.0:1787826426.562682:0:55751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114401 00010000:00010000:3.0:1787826426.562749:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1832 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.562758:0:55751:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1832 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787826426.562771:0:55751:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787826426.562786:0:55751:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff97688cb5d2c0 x1874670564007936/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787826426.562797:0:55751:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1832 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826426.562889:0:55751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564007936:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826426.563260:0:55751:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25540 going to sleep for 11 seconds 00000100:00100000:3.0:1787826426.566433:0:55751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564007936:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826426.566473:0:55751:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1832 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b839ae expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.566492:0:55751:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1832 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b839ae expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.566504:0:55751:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1832 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b839ae expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826426.566522:0:55751:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff97688cb5d2c0 x1874670564007936/t8589935270(8589935270) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826426.566537:0:55751:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff97688cb5d2c0 x1874670564007936/t8589935270(8589935270) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826426.566655:0:55751:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1832 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b839ae expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826426.566672:0:55751:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1832 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b839ae expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.566687:0:55751:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826426.566695:0:55751:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826426.566762:0:55751:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 39 -> 39 00000080:00010000:3.0:1787826426.566768:0:55751:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1832 00000080:00010000:3.0:1787826426.566776:0:55751:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1832 from it ffffb2d4422abbf0 00010000:00010000:3.0:1787826426.566780:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1832 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b839ae expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.566791:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1832 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b839ae expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000002:00100000:3.0:1787826426.566927:0:55751:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff97688cb5d2c0 x1874670564007936/t8589935270(8589935270) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1787826426.567305:0:55751:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:3.0:1787826426.567316:0:55751:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:3.0:1787826426.567360:0:55751:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826426.567368:0:55751:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:3.0:1787826426.567397:0:55751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826426.567411:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826426.567430:0:55751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826426.567447:0:55751:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826426.567454:0:55751:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:3.0:1787826426.567458:0:55751:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1824 00000080:00010000:3.0:1787826426.567464:0:55751:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1824 from it ffffb2d4422abcd8 00010000:00010000:3.0:1787826426.567468:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826426.567477:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826426.567501:0:55751:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1787826426.567515:0:55751:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:3.0:1787826426.567539:0:55751:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1787826426.567543:0:55751:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff9768837ff530, osc lock ffff9768984ef998, flags 0x400001002 00010000:00010000:3.0:1787826426.567564:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826426.567576:0:55751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826426.567585:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826426.567599:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826426.567643:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab1839 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.567651:0:55751:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab1839 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787826426.567704:0:55751:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1787826426.567724:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787826426.567745:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564008064:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826426.570227:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826426.571188:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585293440 00000100:00100000:1.0:1787826426.571201:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585293440 00000100:00100000:1.0:1787826426.571279:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 615 00000100:00100000:1.0:1787826426.571289:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aec780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:14864:x1874670585293440:12345-192.168.203.139@tcp:106: 00010000:00010000:1.0:1787826426.571309:0:53061:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826426.571323:0:53061:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00010000:1.0:1787826426.571370:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@00000000c0eb4eb5[0x110, 1, [0x200000405:0x42:0x0] hash] 00000020:00010000:1.0:1787826426.571373:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826426 blocks: 0 00000020:00010000:1.0:1787826426.571377:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@0000000048e089aa[0x116, 2, [0x280000401:0x10e:0x0]] 00000020:00010000:1.0:1787826426.571379:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826426 blocks: 0 00000100:00100000:1.0:1787826426.571889:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aec780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:14864:x1874670585293440:12345-192.168.203.139@tcp:106: Request processed in 598us (1673us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826426.571901:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 615 00010000:00010000:3.0:1787826426.580273:0:2404:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab1839 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.580304:0:2404:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab1839 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.580322:0:2404:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab1839 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.580330:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab1839 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.580344:0:2404:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab1839 lrc: 5/1,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826426.580693:0:2404:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:3.0:1787826426.580703:0:2404:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826426.580759:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab1839 lrc: 4/1,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.580769:0:2404:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab1839 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.580777:0:2404:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab1839 lrc: 4/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.580784:0:2404:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976888fb9a00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826426.580790:0:2404:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab1839 lrc: 5/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.580803:0:2404:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826426.580810:0:2404:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab1839 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.580818:0:2404:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000778c89cf/0x8e7c99ee62ab1839 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826426.580870:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564008064:192.168.203.139@tcp:101:lfs.0 00000080:00200000:2.0:1787826426.580965:0:55751:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1787826426.580973:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf1da0 (ffff976890d10e18/0) at cl_lock_release():245 00000020:00010000:2.0:1787826426.580977:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf1da0 (ffff976890d10e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787826426.580983:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768837ff530 (ffff976890d10128/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787826426.580988:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf1da0 (ffff976890d10e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1787826426.580991:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768837ff530 (ffff976890d10e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1787826426.581009:0:55751:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 39 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1787826426.581084:0:55751:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154000 00010000:00010000:2.0:1787826426.581124:0:55751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1832 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b839ae expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826426.581267:0:55751:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:2.0:1787826426.581293:0:55751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826426.581307:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826426.581318:0:55751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826426.581326:0:55751:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826426.581335:0:55751:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:2.0:1787826426.581339:0:55751:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:2.0:1787826426.581346:0:55751:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4422abbf8 00010000:00010000:2.0:1787826426.581815:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826426.581976:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826426.581998:0:55751:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:2.0:1787826426.582008:0:55751:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:2.0:1787826426.582016:0:55751:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787826426.582020:0:55751:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:2.0:1787826426.582025:0:55751:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 46 00000080:00200008:2.0:1787826426.582036:0:55751:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 1154402 00000002:00010000:2.0:1787826426.582060:0:55751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114403 00010000:00010000:2.0:1787826426.582125:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1840 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.582135:0:55751:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1840 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1787826426.582154:0:55751:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1787826426.582168:0:55751:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efa940 x1874670564008320/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1787826426.582192:0:55751:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1840 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826426.582220:0:55751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efa940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564008320:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826426.582873:0:55751:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35d80 going to sleep for 11 seconds 00000100:00100000:1.0:1787826426.586103:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826426.586155:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585293568 00000100:00100000:1.0:1787826426.586167:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585293568 00000100:00100000:1.0:1787826426.586239:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 616 00000100:00100000:1.0:1787826426.586249:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0160b40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6518:x1874670585293568:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826426.586272:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787826426.586377:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826426.586385:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0160b40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6518:x1874670585293568:12345-192.168.203.139@tcp:104: Request processed in 138us (299us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826426.586394:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 616 00010000:00010000:2.0:1787826426.586480:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826426.586490:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.586500:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925c400 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:2.0:1787826426.586506:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.586515:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.586524:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.586568:0:42220:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1832 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b839ae expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.586623:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 2/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.586636:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826426.586672:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 1/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.586681:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab1824 lrc: 0/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83984 expref: -99 pid: 55745 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.586702:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.586705:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.586706:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826426.586717:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1787826426.586735:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564008448:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826426.589617:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564008448:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826426.591056:0:55751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efa940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564008320:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826426.591088:0:55751:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode CW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1840 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b839ca expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.591102:0:55751:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1840 lrc: 4/1,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b839ca expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.591117:0:55751:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1840 lrc: 4/1,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b839ca expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.591130:0:55751:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1840 lrc: 4/1,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b839ca expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.591151:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1840 lrc: 5/1,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b839ca expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.591161:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1840 lrc: 5/1,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b839ca expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.591170:0:55751:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1840 lrc: 4/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b839ca expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826426.591193:0:55751:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff976889efa940 x1874670564008320/t8589935271(8589935271) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787826426.591208:0:55751:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efa940 x1874670564008320/t8589935271(8589935271) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826426.591228:0:55751:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1840 lrc: 3/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b839ca expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.591244:0:55751:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 8 res 8589935621/29 (0 0) 00000080:00200000:2.0:1787826426.591251:0:55751:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1787826426.591309:0:55751:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1840 00000080:00010000:2.0:1787826426.591317:0:55751:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1840 from it ffffb2d4422abbf0 00010000:00010000:2.0:1787826426.591321:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1840 lrc: 3/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b839ca expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.591331:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1840 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b839ca expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:2.0:1787826426.591370:0:55751:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efa940 x1874670564008320/t8589935271(8589935271) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:2.0:1787826426.591485:0:55751:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=800466af arg=7ffd1ef205bc 00000080:00200000:2.0:1787826426.591743:0:55751:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:2.0:1787826426.591763:0:55751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826426.591776:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826426.591788:0:55751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826426.591795:0:55751:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826426.591801:0:55751:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:2.0:1787826426.591805:0:55751:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:2.0:1787826426.591809:0:55751:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4422abbf8 00010000:00010000:2.0:1787826426.591812:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826426.591820:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826426.591880:0:55751:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:2.0:1787826426.591888:0:55751:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=24 00000080:00200000:2.0:1787826426.591893:0:55751:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787826426.591909:0:55751:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 3 00000080:00200000:2.0:1787826426.591915:0:55751:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=. :VOLATILE:0000:4E0C50C4:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), file ffff9768837db680, open_flags 2f9c1, mode 8180 opened 0 00000080:00200000:2.0:1787826426.591923:0:55751:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=. :VOLATILE:0000:4E0C50C4:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=open|creat 00000002:00010000:2.0:1787826426.591945:0:55751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: . :VOLATILE:0000:4E0C50C4:fd=03,[0x200000405:0x44:0x0]) in obj [0x200000bd0:0x4:0x0], intent: open|creat flags 0574702 00010000:00010000:2.0:1787826426.591999:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1847 lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.592007:0:55751:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1847 lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1787826426.592020:0:55751:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1787826426.592028:0:55751:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efb480 x1874670564008704/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/66640 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1787826426.592038:0:55751:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1847 lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826426.592061:0:55751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb480 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564008704:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826426.592245:0:55751:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35d80 going to sleep for 11 seconds 00000100:00100000:2.0:1787826426.602934:0:55751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb480 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564008704:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826426.602970:0:55751:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1847 lrc: 4/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b839df expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.603009:0:55751:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1847 lrc: 4/0,1 mode: --/PR res: [0x200000405:0x44:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b839df expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.603019:0:55751:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1847 lrc: 4/0,1 mode: --/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b839df expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.603033:0:55751:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1847 lrc: 4/0,1 mode: PR/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b839df expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.603047:0:55751:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1847 lrc: 4/0,1 mode: PR/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b839df expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.603062:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1847 lrc: 5/1,1 mode: PR/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b839df expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.603071:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1847 lrc: 5/1,1 mode: PR/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b839df expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.603078:0:55751:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1847 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b839df expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826426.603092:0:55751:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff976889efb480 x1874670564008704/t8589935272(8589935272) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787826426.603107:0:55751:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efb480 x1874670564008704/t8589935272(8589935272) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826426.603121:0:55751:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1847 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b839df expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.603133:0:55751:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/68 (0 0) 00000080:00200000:2.0:1787826426.603140:0:55751:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd0:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787826426.603168:0:55751:0:(namei.c:1628:ll_create_it()) VFS Op:name=. :VOLATILE:0000:4E0C50C4:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=open|creat 00000080:00200000:2.0:1787826426.603254:0:55751:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000405:0x44:0x0](ffff97688b7baa08) 00000080:00200000:2.0:1787826426.603263:0:55751:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x44:0x0], updating i_size 0 00000080:00200000:2.0:1787826426.603309:0:55751:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x44:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1787826426.603314:0:55751:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000405:0x44:0x0](ffff97688b7baa08) 00000080:00010000:2.0:1787826426.603319:0:55751:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x44:0x0](ffff97688b7baa08) for lock 0x8e7c99ee62ab1847 00000080:00010000:2.0:1787826426.603331:0:55751:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x44:0x0](ffff97688b7baa08) 00000080:00200008:2.0:1787826426.603360:0:55751:0:(file.c:1090:ll_file_open()) START file . :VOLATILE:0000:4E0C50C4:fd=03:[0x200000405:0x44:0x0](ffff97688b7baa08), flags 574701 00000002:00100000:2.0:1787826426.603390:0:55751:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efb480 x1874670564008704/t8589935272(8589935272) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1787826426.603426:0:55751:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1847 from it ffff9768876c9240 00010000:00010000:2.0:1787826426.603432:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1847 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b839df expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.603916:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1847 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b839df expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826426.603949:0:55751:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x44:0x0](ffff97688b7baa08), inode mode 8180 mask 40 00000080:00200000:2.0:1787826426.604017:0:55751:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x44:0x0](ffff97688b7baa08), inode mode 8180 mask 2 00000080:00200000:2.0:1787826426.604029:0:55751:0:(xattr.c:414:ll_xattr_set()) VFS Op:inode=[0x200000405:0x44:0x0](ffff97688b7baa08), xattr lov 00000002:00010000:2.0:1787826426.604053:0:55751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x44:0x0]) in obj [0x200000405:0x44:0x0], intent: open flags 040000002 00010000:00010000:2.0:1787826426.604121:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab184e lrc: 3/1,0 mode: --/CR res: [0x200000405:0x44:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.604129:0:55751:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab184e lrc: 3/1,0 mode: --/CR res: [0x200000405:0x44:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1787826426.604142:0:55751:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1787826426.604935:0:55751:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889ef92c0 x1874670564008960/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1787826426.604949:0:55751:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab184e lrc: 3/1,0 mode: --/CR res: [0x200000405:0x44:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826426.604982:0:55751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef92c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564008960:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826426.605762:0:55751:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35d80 going to sleep for 11 seconds 00000100:00100000:0.0:1787826426.612052:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826426.612191:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585293696 00000100:00100000:1.0:1787826426.612204:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585293696 00000100:00100000:1.0:1787826426.612312:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 617 00000100:00100000:1.0:1787826426.612324:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aed2c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:8247:x1874670585293696:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826426.612347:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1847 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b839df expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826426.612455:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826426.612462:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aed2c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:8247:x1874670585293696:12345-192.168.203.139@tcp:104: Request processed in 141us (426us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826426.612476:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 617 00010000:00010000:2.0:1787826426.612559:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826426.612568:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1847 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b839df expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.612579:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514c00 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:2.0:1787826426.612586:0:42221:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1847 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b839df expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.612596:0:42221:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1847 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b839df expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.612605:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1847 lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b839df expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826426.612636:0:42221:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x44:0x0]: losing layout lock 00010000:00010000:2.0:1787826426.612697:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1847 lrc: 2/0,0 mode: --/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b839df expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.612704:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826426.612734:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1847 lrc: 1/0,0 mode: --/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b839df expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.612743:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1847 lrc: 0/0,0 mode: --/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b839df expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.612763:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.612765:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.612767:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826426.612812:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564009088:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826426.618029:0:55751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef92c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564008960:192.168.203.139@tcp:101:lfs.0 00000100:00100000:3.0:1787826426.618045:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564009088:192.168.203.139@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1787826426.618066:0:55751:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab184e lrc: 4/1,0 mode: --/CR res: [0x200000405:0x44:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.618085:0:55751:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab184e lrc: 4/1,0 mode: --/CR res: [0x200000405:0x44:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.618093:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab184e lrc: 4/1,0 mode: --/CR res: [0x200000405:0x44:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.618101:0:55751:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab184e lrc: 3/0,0 mode: --/CR res: [0x200000405:0x44:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.618110:0:55751:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab184e lrc: 4/0,0 mode: --/CR res: [0x200000405:0x44:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.618117:0:55751:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514a00 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:2.0:1787826426.618123:0:55751:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab184e lrc: 4/0,0 mode: --/CR res: [0x200000405:0x44:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.618129:0:55751:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab184e lrc: 4/0,0 mode: --/CR res: [0x200000405:0x44:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.618137:0:55751:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab184e lrc: 5/0,0 mode: --/CR res: [0x200000405:0x44:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.618161:0:55751:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1787826426.618169:0:55751:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab184e lrc: 3/0,0 mode: --/CR res: [0x200000405:0x44:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.618185:0:55751:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab184e lrc: 0/0,0 mode: --/CR res: [0x200000405:0x44:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826426.618216:0:55751:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x800002b, status=0 req@ffff976889ef92c0 x1874670564008960/t8589935273(8589935273) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787826426.618230:0:55751:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889ef92c0 x1874670564008960/t8589935273(8589935273) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1787826426.618250:0:55751:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x44:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787826426.618305:0:55751:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x44:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787826426.618315:0:55751:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x44:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787826426.618630:0:55751:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x44:0x0], updating i_size 0 00000002:00100000:2.0:1787826426.618655:0:55751:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889ef92c0 x1874670564008960/t8589935273(8589935273) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787826426.618699:0:55751:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889ef92c0 x1874670564008960/t8589935273(8589935273) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787826426.618711:0:55751:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787826426.618716:0:55751:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889ef8000 x1874670564009344/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787826426.618741:0:55751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889ef8000 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564009344:192.168.203.139@tcp:35:lfs.0 00000100:00100000:2.0:1787826426.618885:0:55751:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35c00 going to sleep for 11 seconds 00000100:00100000:2.0:1787826426.625791:0:55751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889ef8000 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564009344:192.168.203.139@tcp:35:lfs.0 00000002:00100000:2.0:1787826426.625887:0:55751:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889ef92c0 x1874670564008960/t8589935273(8589935273) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826442 ref 2 fl Complete:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1787826426.625941:0:55751:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787826426.625952:0:55751:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:2.0:1787826426.625976:0:55751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826426.625995:0:55751:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 30 res 8589935621/29 (0 0) 00010000:00010000:2.0:1787826426.626065:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.626076:0:55751:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.626088:0:55751:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826426.626116:0:55751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564009472:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826426.626234:0:55751:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826426.630468:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826426.630695:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585293824 00000100:00100000:1.0:1787826426.630709:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585293824 00000100:00100000:1.0:1787826426.630788:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 618 00000100:00100000:1.0:1787826426.630801:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aecb40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:32404:x1874670585293824:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826426.630873:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1840 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x22153cd396b839ca expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826426.630960:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826426.630968:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aecb40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:32404:x1874670585293824:12345-192.168.203.139@tcp:104: Request processed in 170us (516us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826426.630976:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 618 00010000:00010000:2.0:1787826426.631798:0:49796:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826426.631810:0:49796:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1840 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x22153cd396b839ca expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.631850:0:49796:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514200 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:2.0:1787826426.631857:0:49796:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1840 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x22153cd396b839ca expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.631866:0:49796:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1840 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x22153cd396b839ca expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.631882:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1840 lrc: 3/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x22153cd396b839ca expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.631919:0:49796:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1832 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b839ae expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826426.631974:0:49796:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1840 lrc: 2/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x22153cd396b839ca expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.631981:0:49796:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826426.632016:0:49796:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1840 lrc: 1/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x22153cd396b839ca expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.632025:0:49796:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514200/0x8e7c99ee62ab1840 lrc: 0/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x5 rrc: 5 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x22153cd396b839ca expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.632044:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.632045:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.632047:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826426.632187:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826426.632212:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564009600:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826426.636004:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564009600:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826426.638920:0:55751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564009472:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826426.638955:0:55751:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.638968:0:55751:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.638983:0:55751:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.638996:0:55751:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.639011:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.639020:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.639028:0:55751:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826426.639041:0:55751:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff976889efbc00 x1874670564009472/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 576/720 e 0 to 0 dl 1787826442 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826426.639055:0:55751:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787826426.639071:0:55751:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.639086:0:55751:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/29 (0 0) 00000080:00200000:2.0:1787826426.639093:0:55751:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787826426.639160:0:55751:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 39 -> 39 00000080:00010000:2.0:1787826426.639166:0:55751:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1855 00000080:00010000:2.0:1787826426.639182:0:55751:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:2.0:1787826426.639186:0:55751:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1855 from it ffffb2d4422abcd8 00010000:00010000:2.0:1787826426.639190:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.639200:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826426.639239:0:55751:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 39 restore needed 0 00000001:00010000:2.0:1787826426.639253:0:55751:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:2.0:1787826426.639277:0:55751:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787826426.639281:0:55751:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff97688757c9b0, osc lock ffff976883f7a270, flags 0x400001002 00010000:00010000:2.0:1787826426.639303:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.639319:0:55751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.639329:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.639337:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826426.639381:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000023579fe9/0x8e7c99ee62ab185c lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.639390:0:55751:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000023579fe9/0x8e7c99ee62ab185c lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1787826426.639438:0:55751:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1787826426.639990:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787826426.640020:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564009728:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826426.643636:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826426.643868:0:53061:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1787826426.643894:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585293952 00000100:00100000:1.0:1787826426.643906:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585293952 00000100:00100000:1.0:1787826426.643968:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 619 00000100:00100000:1.0:1787826426.643979:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff97688a33cc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:30694:x1874670585293952:12345-192.168.203.139@tcp:106: 00010000:00010000:1.0:1787826426.644000:0:53061:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787826426.644015:0:53061:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00000020:00010000:1.0:1787826426.644045:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@00000000c0eb4eb5[0x110, 1, [0x200000405:0x42:0x0] hash] 00000020:00010000:1.0:1787826426.644048:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826426 blocks: 0 00000020:00010000:1.0:1787826426.644052:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@0000000048e089aa[0x116, 2, [0x280000401:0x10e:0x0]] 00000020:00010000:1.0:1787826426.644054:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826426 blocks: 0 00000100:00100000:1.0:1787826426.644122:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff97688a33cc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:30694:x1874670585293952:12345-192.168.203.139@tcp:106: Request processed in 144us (502us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826426.644131:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 619 00010000:00010000:3.0:1787826426.646935:0:2404:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000023579fe9/0x8e7c99ee62ab185c lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.646970:0:2404:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000023579fe9/0x8e7c99ee62ab185c lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.646986:0:2404:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000023579fe9/0x8e7c99ee62ab185c lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.646994:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000023579fe9/0x8e7c99ee62ab185c lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.647074:0:2404:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000023579fe9/0x8e7c99ee62ab185c lrc: 5/1,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826426.647092:0:2404:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:3.0:1787826426.647101:0:2404:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826426.647321:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000023579fe9/0x8e7c99ee62ab185c lrc: 4/1,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.647331:0:2404:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000023579fe9/0x8e7c99ee62ab185c lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.647340:0:2404:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000023579fe9/0x8e7c99ee62ab185c lrc: 4/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.647348:0:2404:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514200 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826426.647354:0:2404:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000023579fe9/0x8e7c99ee62ab185c lrc: 5/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.647367:0:2404:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826426.647376:0:2404:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000023579fe9/0x8e7c99ee62ab185c lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.647385:0:2404:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 0000000023579fe9/0x8e7c99ee62ab185c lrc: 0/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826426.647411:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564009728:192.168.203.139@tcp:101:lfs.0 00000080:00200000:2.0:1787826426.647640:0:55751:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1787826426.647649:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf1da0 (ffff976890d10e18/0) at cl_lock_release():245 00000020:00010000:2.0:1787826426.647653:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf1da0 (ffff976890d10e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787826426.647659:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757c9b0 (ffff976890d10128/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787826426.647663:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf1da0 (ffff976890d10e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1787826426.647666:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757c9b0 (ffff976890d10e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1787826426.647680:0:55751:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 39 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1787826426.647709:0:55751:0:(file.c:6313:ll_getattr_dentry()) START file . :VOLATILE:0000:4E0C50C4:fd=03:[0x200000405:0x44:0x0](ffff97688b7baa08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787826426.647718:0:55751:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x44:0x0](ffff97688b7baa08),name=. :VOLATILE:0000:4E0C50C4:fd=03 00000002:00010000:2.0:1787826426.647744:0:55751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x44:0x0]) in obj [0x200000405:0x44:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826426.647809:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1863 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x44:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.647819:0:55751:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1863 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x44:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.648082:0:55751:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1863 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x44:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826426.648108:0:55751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564009856:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826426.648298:0:55751:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35b40 going to sleep for 11 seconds 00000100:00100000:2.0:1787826426.651679:0:55751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564009856:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826426.651716:0:55751:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1863 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x44:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a1e expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.651743:0:55751:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1863 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a1e expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.651757:0:55751:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1863 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83a1e expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.651770:0:55751:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1863 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a1e expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.651786:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1863 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83a1e expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.651794:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1863 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83a1e expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.651802:0:55751:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1863 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83a1e expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826426.651817:0:55751:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff976889efbc00 x1874670564009856/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 576/720 e 0 to 0 dl 1787826442 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826426.651988:0:55751:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1863 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a1e expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787826426.652072:0:55751:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1863 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a1e expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.652094:0:55751:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/68 (0 0) 00000080:00200000:2.0:1787826426.652102:0:55751:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000405:0x44:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787826426.652260:0:55751:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x44:0x0]: layout version change: 4294967294 -> 1 00000080:00010000:2.0:1787826426.652266:0:55751:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x44:0x0](ffff97688b7baa08) for lock 0x8e7c99ee62ab1863 00000080:00010000:2.0:1787826426.652272:0:55751:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x44:0x0](ffff97688b7baa08) 00000080:00010000:2.0:1787826426.652275:0:55751:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1863 from it ffffb2d4422abcd8 00010000:00010000:2.0:1787826426.652279:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1863 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83a1e expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.652290:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1863 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83a1e expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826426.652333:0:55751:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x44:0x0] ignore/verify layout 0/1, layout version 39 restore needed 0 00000001:00010000:2.0:1787826426.652348:0:55751:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x44:0x0] 00000008:00010000:2.0:1787826426.652372:0:55751:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787826426.652376:0:55751:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff97688757cb30, osc lock ffff976883f7a270, flags 0x400001002 00010000:00010000:2.0:1787826426.652437:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab186a lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.652448:0:55751:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab186a lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826426.652580:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564009984:192.168.203.139@tcp:101:lfs.0 00000020:00200000:2.0:1787826426.652964:0:55751:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1787826426.656322:0:2404:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab186a lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.656533:0:2404:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab186a lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a25 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.656670:0:2404:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab186a lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83a25 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.656685:0:2404:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab186a lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a25 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826426.656705:0:2404:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab186a lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83a25 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787826426.656719:0:2404:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826426.656740:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab186a lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83a25 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.656750:0:2404:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab186a lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83a25 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.656758:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab186a lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83a25 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.656773:0:2404:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab186a lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83a25 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787826426.656796:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564009984:192.168.203.139@tcp:101:lfs.0 00000080:00200000:2.0:1787826426.656956:0:55751:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x44:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1787826426.656965:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf1da0 (ffff976890d10e18/0) at cl_lock_release():245 00000020:00010000:2.0:1787826426.656969:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf1da0 (ffff976890d10e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787826426.656975:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757cb30 (ffff976890d10128/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787826426.656980:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf1da0 (ffff976890d10e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1787826426.656984:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757cb30 (ffff976890d10e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1787826426.657002:0:55751:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x44:0x0] ignore/verify layout 0/1, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200080:2.0:1787826426.657035:0:55751:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=7ffd1ef22704 00000002:00010000:2.0:1787826426.657073:0:55751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 01004040000001 00010000:00010000:2.0:1787826426.657146:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1871 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.657155:0:55751:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x81000000001000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1871 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1787826426.657169:0:55751:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1787826426.657189:0:55751:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efb0c0 x1874670564010240/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1787826426.657199:0:55751:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1871 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x81000000000000 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826426.657226:0:55751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564010240:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826426.658202:0:55751:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e359c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787826426.661958:0:55751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efb0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564010240:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826426.661998:0:55751:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1871 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83a33 expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.662013:0:55751:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1871 lrc: 4/1,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83a33 expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.662030:0:55751:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1871 lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83a33 expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.662044:0:55751:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1871 lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83a33 expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.662061:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1871 lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83a33 expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.662071:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1871 lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83a33 expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.662078:0:55751:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1871 lrc: 4/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83a33 expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826426.662092:0:55751:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x600002b, status=0 req@ffff976889efb0c0 x1874670564010240/t8589935275(8589935275) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787826426.662107:0:55751:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efb0c0 x1874670564010240/t8589935275(8589935275) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/744 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826426.662127:0:55751:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1871 lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83a33 expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.662146:0:55751:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 2 res 8589935621/29 (0 0) 00000080:00200000:2.0:1787826426.662155:0:55751:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000002:00100000:2.0:1787826426.662197:0:55751:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889efb0c0 x1874670564010240/t8589935275(8589935275) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826442 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1787826426.662206:0:55751:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1871 00000080:00010000:2.0:1787826426.662214:0:55751:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1871 from it ffffb2d4422abcc0 00010000:00010000:2.0:1787826426.662218:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1871 lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83a33 expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.662225:0:55751:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1871 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83a33 expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200008:2.0:1787826426.662300:0:55751:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787826426.662310:0:55751:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:2.0:1787826426.662331:0:55751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826426.662343:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.662369:0:55751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826426.662379:0:55751:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826426.662386:0:55751:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:2.0:1787826426.662390:0:55751:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1855 00000080:00010000:2.0:1787826426.662395:0:55751:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1855 from it ffffb2d4422abcd8 00010000:00010000:2.0:1787826426.662398:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.662408:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826426.662760:0:55751:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 1 restore needed 0 00000001:00010000:2.0:1787826426.662776:0:55751:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:2.0:1787826426.662803:0:55751:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787826426.662808:0:55751:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff97688757cb30, osc lock ffff976883f7a270, flags 0x400001002 00010000:00010000:2.0:1787826426.662885:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826426.662899:0:55751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826426.662910:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826426.662918:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826426.662968:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cf3c1d8e/0x8e7c99ee62ab1878 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.662977:0:55751:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cf3c1d8e/0x8e7c99ee62ab1878 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1787826426.663024:0:55751:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1787826426.663693:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564010368:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826426.666585:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826426.667118:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585294080 00000100:00100000:1.0:1787826426.667132:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585294080 00000100:00100000:1.0:1787826426.667566:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 620 00000100:00100000:1.0:1787826426.667580:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aee580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30398:x1874670585294080:12345-192.168.203.139@tcp:106: 00010000:00010000:1.0:1787826426.667600:0:42210:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1787826426.667615:0:42210:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00000020:00010000:1.0:1787826426.667648:0:42210:0:(cl_object.c:322:cl_object_glimpse()) header@00000000c0eb4eb5[0x110, 1, [0x200000405:0x42:0x0] hash] 00000020:00010000:1.0:1787826426.667651:0:42210:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826426 blocks: 0 00000020:00010000:1.0:1787826426.667655:0:42210:0:(cl_object.c:322:cl_object_glimpse()) header@0000000048e089aa[0x116, 2, [0x280000401:0x10e:0x0]] 00000020:00010000:1.0:1787826426.667657:0:42210:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826426 blocks: 0 00000100:00100000:1.0:1787826426.667739:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aee580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30398:x1874670585294080:12345-192.168.203.139@tcp:106: Request processed in 161us (1166us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826426.667747:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 620 00010000:00010000:3.0:1787826426.672050:0:2403:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cf3c1d8e/0x8e7c99ee62ab1878 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.672080:0:2403:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cf3c1d8e/0x8e7c99ee62ab1878 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.672096:0:2403:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cf3c1d8e/0x8e7c99ee62ab1878 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.672104:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cf3c1d8e/0x8e7c99ee62ab1878 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.672113:0:2403:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cf3c1d8e/0x8e7c99ee62ab1878 lrc: 5/1,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826426.672128:0:2403:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:3.0:1787826426.672136:0:2403:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826426.672156:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cf3c1d8e/0x8e7c99ee62ab1878 lrc: 4/1,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.672166:0:2403:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cf3c1d8e/0x8e7c99ee62ab1878 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.672182:0:2403:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cf3c1d8e/0x8e7c99ee62ab1878 lrc: 4/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.672189:0:2403:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515200 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826426.672196:0:2403:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cf3c1d8e/0x8e7c99ee62ab1878 lrc: 5/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.672209:0:2403:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826426.672218:0:2403:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cf3c1d8e/0x8e7c99ee62ab1878 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.672227:0:2403:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000cf3c1d8e/0x8e7c99ee62ab1878 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826426.672251:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564010368:192.168.203.139@tcp:101:lfs.0 00000080:00200000:2.0:1787826426.672310:0:55751:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1787826426.672318:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf1da0 (ffff976890d10e18/0) at cl_lock_release():245 00000020:00010000:2.0:1787826426.672326:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf1da0 (ffff976890d10e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787826426.672333:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757cb30 (ffff976890d10128/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787826426.672337:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf1da0 (ffff976890d10e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1787826426.672340:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757cb30 (ffff976890d10e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1787826426.672357:0:55751:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 39 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1787826426.672389:0:55751:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787826426.672398:0:55751:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:2.0:1787826426.672425:0:55751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826426.672446:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.672460:0:55751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826426.672558:0:55751:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826426.672568:0:55751:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:2.0:1787826426.672573:0:55751:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1855 00000080:00010000:2.0:1787826426.672580:0:55751:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1855 from it ffffb2d4422abcd8 00010000:00010000:2.0:1787826426.672584:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.672593:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787826426.672619:0:55751:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 39 restore needed 0 00000001:00010000:2.0:1787826426.672630:0:55751:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:2.0:1787826426.672651:0:55751:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787826426.672655:0:55751:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff97688757cb30, osc lock ffff976883f7a270, flags 0x400001002 00010000:00010000:2.0:1787826426.672674:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826426.672684:0:55751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826426.672694:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826426.672701:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1787826426.672743:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab187f lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.672752:0:55751:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab187f lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826426.674645:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564010496:192.168.203.139@tcp:101:lfs.0 00000020:00200000:2.0:1787826426.675603:0:55751:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1787826426.683246:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826426.683375:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585294208 00000100:00100000:1.0:1787826426.683389:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585294208 00000100:00100000:1.0:1787826426.683737:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 621 00000100:00100000:1.0:1787826426.683751:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aef0c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:8545:x1874670585294208:12345-192.168.203.139@tcp:106: 00010000:00010000:1.0:1787826426.683773:0:42211:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1787826426.683788:0:42211:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00000020:00010000:1.0:1787826426.683873:0:42211:0:(cl_object.c:322:cl_object_glimpse()) header@00000000c0eb4eb5[0x110, 1, [0x200000405:0x42:0x0] hash] 00000020:00010000:1.0:1787826426.683876:0:42211:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826426 blocks: 0 00000020:00010000:1.0:1787826426.683880:0:42211:0:(cl_object.c:322:cl_object_glimpse()) header@0000000048e089aa[0x116, 2, [0x280000401:0x10e:0x0]] 00000020:00010000:1.0:1787826426.683882:0:42211:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826426 blocks: 0 00000100:00100000:1.0:1787826426.683953:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aef0c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:8545:x1874670585294208:12345-192.168.203.139@tcp:106: Request processed in 206us (725us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826426.683962:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 621 00010000:00010000:3.0:1787826426.692324:0:2404:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab187f lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.692458:0:2404:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab187f lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.692730:0:2404:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab187f lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.692740:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab187f lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.692749:0:2404:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab187f lrc: 5/1,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826426.692765:0:2404:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:3.0:1787826426.692773:0:2404:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826426.692792:0:2404:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab187f lrc: 4/1,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.692799:0:2404:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab187f lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.692809:0:2404:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab187f lrc: 4/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.692818:0:2404:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514e00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:3.0:1787826426.692988:0:2404:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab187f lrc: 5/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.693009:0:2404:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826426.693018:0:2404:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab187f lrc: 3/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.693027:0:2404:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000007aba44f2/0x8e7c99ee62ab187f lrc: 0/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826426.693052:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564010496:192.168.203.139@tcp:101:lfs.0 00000080:00200000:2.0:1787826426.693109:0:55751:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1787826426.693117:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf1da0 (ffff976890d10e18/0) at cl_lock_release():245 00000020:00010000:2.0:1787826426.693126:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf1da0 (ffff976890d10e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787826426.693133:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757cb30 (ffff976890d10128/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787826426.693137:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf1da0 (ffff976890d10e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1787826426.693140:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757cb30 (ffff976890d10e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1787826426.693165:0:55751:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 39 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200080:2.0:1787826426.693201:0:55751:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=801066da arg=7ffd1ef22600 00000080:00200000:2.0:1787826426.693214:0:55751:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 39 restore needed 0 00020000:00200000:2.0:1787826426.693228:0:55751:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1787826426.693405:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564010624:192.168.203.139@tcp:1:lfs.0 00000100:00100000:0.0:1787826426.697352:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826426.697487:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585294336 00000100:00100000:1.0:1787826426.697500:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585294336 00000100:00100000:1.0:1787826426.697790:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 622 00000100:00100000:1.0:1787826426.697804:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aecf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:30696:x1874670585294336:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826426.697923:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1787826426.698140:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826426.698149:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aecf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:30696:x1874670585294336:12345-192.168.203.139@tcp:104: Request processed in 348us (811us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826426.698166:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 622 00010000:00010000:2.0:1787826426.698340:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826426.698349:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1787826426.698361:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515000 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:2.0:1787826426.698368:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400020000 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1787826426.698398:0:42220:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x42:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787826426.698415:0:42220:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x42:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1787826426.698472:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1787826426.698481:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826426.698507:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 1/0,0 mode: --/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1787826426.698516:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab17d7 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x22153cd396b8387a expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1787826426.698541:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.698543:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.698545:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826426.698620:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826426.698638:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564010752:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826426.703145:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564010752:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826426.703205:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564010624:192.168.203.139@tcp:1:lfs.0 00000080:00200000:2.0:1787826426.703727:0:55751:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:2.0:1787826426.703740:0:55751:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:2.0:1787826426.703774:0:55751:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787826426.703784:0:55751:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:2.0:1787826426.703811:0:55751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826426.703923:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.703941:0:55751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787826426.703950:0:55751:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826426.703958:0:55751:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:2.0:1787826426.703963:0:55751:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab1855 00000080:00010000:2.0:1787826426.703972:0:55751:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1855 from it ffffb2d4422abcd8 00010000:00010000:2.0:1787826426.703976:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.703985:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1787826426.704014:0:55751:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 39 restore needed 0 00000001:00010000:2.0:1787826426.704026:0:55751:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:2.0:1787826426.704049:0:55751:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787826426.704053:0:55751:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff97688757cb30, osc lock ffff976883f7a270, flags 0x400001002 00010000:00010000:2.0:1787826426.704108:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.704120:0:55751:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826426.704271:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564010880:192.168.203.139@tcp:101:lfs.0 00000020:00200000:2.0:1787826426.704404:0:55751:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1787826426.713075:0:2403:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.713111:0:2403:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.713130:0:2403:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.713146:0:2403:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826426.713235:0:2403:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787826426.713250:0:2403:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826426.713272:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.713283:0:2403:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.713291:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.713301:0:2403:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787826426.713326:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564010880:192.168.203.139@tcp:101:lfs.0 00000080:00200000:2.0:1787826426.713438:0:55751:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1787826426.713447:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976885cf1da0 (ffff976890d10e18/0) at cl_lock_release():245 00000020:00010000:2.0:1787826426.713451:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976885cf1da0 (ffff976890d10e18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787826426.713458:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757cb30 (ffff976890d10128/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787826426.713463:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976885cf1da0 (ffff976890d10e18/0) at cl_lock_fini():72 00000020:00010000:2.0:1787826426.713531:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757cb30 (ffff976890d10e18/1) at cl_lock_fini():72 00000080:00200000:2.0:1787826426.713620:0:55751:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 39 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1787826426.787556:0:55751:0:(file.c:5224:ll_file_seek()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0], offset: 0, type: (null) 00000080:00200000:3.0:1787826426.787580:0:55751:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1787826426.787592:0:55751:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:3.0:1787826426.787626:0:55751:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1787826426.787631:0:55751:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff9768837ff930, osc lock ffff9768984efc08, flags 0x400001002 00010000:00010000:3.0:1787826426.787661:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826426.787690:0:55751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:3.0:1787826426.787707:0:55751:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1787826426.787713:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826426.787724:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:3.0:1787826426.787741:0:55751:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1787826426.787753:0:55751:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:3.0:1787826426.787757:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9768983a0000 (ffff976883b9d518/0) at cl_lock_release():245 00000020:00010000:3.0:1787826426.787761:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768983a0000 (ffff976883b9d518/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826426.787764:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768837ff930 (ffff976883b9d998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1787826426.787769:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768983a0000 (ffff976883b9d518/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826426.787771:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768837ff930 (ffff976883b9d518/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826426.787788:0:55751:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 39 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1787826426.787797:0:55751:0:(file.c:5262:ll_file_seek()) COMPLETED file f60b.sanity-flr:[0x200000405:0x1d:0x0], offset: 0, type: (null), rc = 0 00000080:00200008:3.0:1787826426.787802:0:55751:0:(file.c:5224:ll_file_seek()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0], offset: 0, type: SEEK_HOLE 00000080:00200000:3.0:1787826426.787808:0:55751:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 39 restore needed 0 00020000:00200000:3.0:1787826426.787815:0:55751:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826426.787874:0:55751:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000080:00200000:3.0:1787826426.787884:0:55751:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826426.787890:0:55751:0:(file.c:1848:ll_io_set_mirror()) f60b.sanity-flr: desiginated mirror: 0 00000080:00200000:3.0:1787826426.787893:0:55751:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826426.787898:0:55751:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826426.787903:0:55751:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 18446744073709551615] 00000008:00010000:3.0:1787826426.787910:0:55751:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 8 00000008:00010000:3.0:1787826426.787912:0:55751:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff9768837ff930, osc lock ffff9768984efc08, flags 0x0 00010000:00010000:3.0:1787826426.787921:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826426.787932:0:55751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826426.787942:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826426.787951:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826426.787959:0:55751:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826426.787970:0:55751:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:3.0:1787826426.787976:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976891f52b78 (ffff976883b9d518/0) at cl_lock_release():245 00000020:00010000:3.0:1787826426.787979:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976891f52b78 (ffff976883b9d518/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826426.787981:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768837ff930 (ffff976883b9d998/1) at cl_lock_cancel():142 00010000:00010000:3.0:1787826426.787985:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 4/1,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826426.787994:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00010000:3.0:1787826426.788004:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976891f52b78 (ffff976883b9d518/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826426.788006:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768837ff930 (ffff976883b9d518/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826426.788013:0:55751:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 39 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1787826426.788017:0:55751:0:(file.c:5262:ll_file_seek()) COMPLETED file f60b.sanity-flr:[0x200000405:0x1d:0x0], offset: 0, type: SEEK_HOLE, rc = -6 00000080:00200080:3.0:1787826426.788059:0:55751:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=66f4 arg=55b 00000080:00200008:3.0:1787826426.788079:0:55751:0:(file.c:2485:do_file_read_iter()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0], ppos: 0, count: 67108864 00000080:00200000:3.0:1787826426.788091:0:55751:0:(file.c:2090:ll_file_io_generic()) f60b.sanity-flr: read ppos: 0, bytes: 67108864 00000080:00200000:3.0:1787826426.788099:0:55751:0:(file.c:1848:ll_io_set_mirror()) f60b.sanity-flr: desiginated mirror: 0 00000020:00200000:3.0:1787826426.788106:0:55751:0:(cl_io.c:218:cl_io_rw_init()) header@00000000a0ca5d5e[0x110, 2, [0x200000405:0x1d:0x0] hash] 00000020:00200000:3.0:1787826426.788109:0:55751:0:(cl_io.c:218:cl_io_rw_init()) io range: 1 [0, 67108864) 2048 0 00000080:00200000:3.0:1787826426.788111:0:55751:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 39 restore needed 0 00000080:00200000:3.0:1787826426.788122:0:55751:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 16383] 00020000:00200000:3.0:1787826426.788132:0:55751:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 67108864) 67108864, 67108864 00020000:00200000:3.0:1787826426.788135:0:55751:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826426.788146:0:55751:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 16383] 00000008:00010000:3.0:1787826426.788153:0:55751:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 11 00000008:00010000:3.0:1787826426.788157:0:55751:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff9768837ff930, osc lock ffff9768984efc08, flags 0x40000 00000080:00200000:3.0:1787826426.788163:0:55751:0:(vvp_io.c:874:vvp_io_read_start()) f60b.sanity-flr: read [0, 67108864) 00000001:00010000:3.0:1787826426.788169:0:55751:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:3.0:1787826426.790894:0:55751:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1787826426.790908:0:55751:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff9768837ff4b0, osc lock ffff9768984eec30, flags 0x400001002 00010000:00010000:3.0:1787826426.790946:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826426.790972:0:55751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00200000:3.0:1787826426.790989:0:55751:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1787826426.790994:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826426.791003:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00000020:00200000:3.0:1787826426.791022:0:55751:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1787826426.791035:0:55751:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:3.0:1787826426.791040:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9768983a0000 (ffff976883b9d518/0) at cl_lock_release():245 00000020:00010000:3.0:1787826426.791043:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768983a0000 (ffff976883b9d518/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826426.791046:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768837ff4b0 (ffff976883b9d998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1787826426.791050:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768983a0000 (ffff976883b9d518/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826426.791052:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768837ff4b0 (ffff976883b9d518/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826426.791065:0:55751:0:(vvp_io.c:932:vvp_io_read_start()) count:67108864 iocb pos:0 00000080:00200000:3.0:1787826426.791073:0:55751:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), size=67108864 (max 335544320), offset=0=0x0, pages 16384 (max 81920), lockless, parallel 00000020:00010000:3.0:1787826426.791088:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976891f52b78 (ffff976883b9d518/0) at cl_lock_release():245 00000020:00010000:3.0:1787826426.791091:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976891f52b78 (ffff976883b9d518/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826426.791094:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768837ff930 (ffff976883b9d998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1787826426.791097:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976891f52b78 (ffff976883b9d518/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826426.791099:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768837ff930 (ffff976883b9d518/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826426.791106:0:55751:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000020:00200000:3.0:1787826426.791114:0:55751:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1787826426.791118:0:55751:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 16383] 00000080:00200000:3.0:1787826426.791128:0:55751:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 39 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826426.791133:0:55751:0:(file.c:2240:ll_file_io_generic()) f60b.sanity-flr: 1 io complete with rc: 0, result: 0, restart: 0 00000020:00200000:3.0:1787826426.791137:0:55751:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1787826426.791144:0:55751:0:(file.c:2326:ll_file_io_generic()) iot: 1, result: 0 00000080:00200008:3.0:1787826426.791171:0:55751:0:(file.c:5401:ll_fsync()) START file: name=. :VOLATILE:0000:4E0C50C4:fd=03, fid=[0x200000405:0x44:0x0], start=0, end=9223372036854775807, datasync=0, need_sync 1/0 00000100:00100000:3.0:1787826426.791265:0:55751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564011008:192.168.203.139@tcp:44:lfs.0 00000100:00100000:3.0:1787826426.791409:0:55751:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25900 going to sleep for 11 seconds 00000100:00100000:3.0:1787826426.804277:0:55751:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9768877a4800: committing for last_committed 8589935275 gen 1 00000100:00100000:3.0:1787826426.804338:0:55751:0:(client.c:3174:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff97688cb5d2c0 x1874670564007936/t8589935270(8589935270) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826442 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1787826426.804351:0:55751:0:(client.c:3174:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff976889efa940 x1874670564008320/t8589935271(8589935271) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826442 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1787826426.804360:0:55751:0:(client.c:3174:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff976889efb480 x1874670564008704/t8589935272(8589935272) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826442 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1787826426.804422:0:55751:0:(client.c:3174:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff976889efb0c0 x1874670564010240/t8589935275(8589935275) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826442 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1787826426.804621:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826426.804690:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585294592 00000100:00100000:0.0:1787826426.804708:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585294592 00000100:00100000:0.0:1787826426.804766:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 623 00000100:00100000:0.0:1787826426.804778:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0160b40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:8437:x1874670585294592:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826426.804804:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000030000 nid: local remote: 0x22153cd396b83888 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826426.804932:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826426.804940:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0160b40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:8437:x1874670585294592:12345-192.168.203.139@tcp:104: Request processed in 164us (321us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826426.804947:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 623 00010000:00010000:2.0:1787826426.805089:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826426.805099:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000030000 nid: local remote: 0x22153cd396b83888 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.805110:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514000 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:2.0:1787826426.805117:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400030000 nid: local remote: 0x22153cd396b83888 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.805183:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 2/0,0 mode: --/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400030020 nid: local remote: 0x22153cd396b83888 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.805191:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826426.805218:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 1/0,0 mode: --/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400030020 nid: local remote: 0x22153cd396b83888 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.805227:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab17de lrc: 0/0,0 mode: --/PW res: [0x2c0000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400030020 nid: local remote: 0x22153cd396b83888 expref: -99 pid: 55735 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.805255:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.805257:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.805259:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826426.805276:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564011136:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826426.805563:0:55751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564011008:192.168.203.139@tcp:44:lfs.0 00000080:00200000:3.0:1787826426.805598:0:55751:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x44:0x0] ignore/verify layout 0/0, layout version 39 restore needed 0 00020000:00200000:2.0:1787826426.805781:0:55751:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:2.0:1787826426.805892:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1787826426.805903:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564011264:192.168.203.139@tcp:16:lfs.0 00000100:00100000:2.0:1787826426.810722:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564011264:192.168.203.139@tcp:16:lfs.0 00000080:00200000:3.0:1787826426.810738:0:55751:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x44:0x0] updating i_size 0 i_blocks 0 00000080:00200000:3.0:1787826426.810759:0:55751:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x44:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1787826426.811112:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564011136:192.168.203.139@tcp:103:ldlm_bl.0 00000080:00200080:2.0:1787826426.826623:0:55751:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=801066da arg=7ffd1ef22600 00000080:00200000:2.0:1787826426.826655:0:55751:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 37 restore needed 0 00020000:00200000:2.0:1787826426.826672:0:55751:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1787826426.826853:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564011392:192.168.203.139@tcp:1:lfs.0 00000100:00100000:3.0:1787826426.832367:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564011392:192.168.203.139@tcp:1:lfs.0 00000080:00200008:2.0:1787826426.832703:0:55751:0:(llite_lib.c:2427:ll_setattr_raw()) START file . :VOLATILE:0000:4E0C50C4:fd=03:[0x200000405:0x44:0x0](ffff97688b7baa08) current size 0, valid attrs 101f0, mode 0, uid 0, gid 0, new size 4294967296, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826426.000000000 00010000:00010000:2.0:1787826426.832756:0:55751:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1863 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b83a1e expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826426.832787:0:55751:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x44:0x0]: losing layout lock 00010000:00010000:2.0:1787826426.832950:0:55751:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1863 lrc: 1/0,0 mode: --/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b83a1e expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.832959:0:55751:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826426.832962:0:55751:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514a00/0x8e7c99ee62ab1863 lrc: 0/0,0 mode: --/PR res: [0x200000405:0x44:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b83a1e expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1787826426.832994:0:55751:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787826426.833018:0:55751:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889cf8f00 x1874670564011648/t0(0) o36->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826426.833049:0:55751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564011648:192.168.203.139@tcp:36:lfs.0 00000100:00100000:2.0:1787826426.833216:0:55751:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35b40 going to sleep for 11 seconds 00000100:00100000:2.0:1787826426.837628:0:55751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564011648:192.168.203.139@tcp:36:lfs.0 00000080:00200000:2.0:1787826426.837699:0:55751:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x44:0x0] ignore/verify layout 0/0, layout version 39 restore needed 0 00000080:00010000:2.0:1787826426.837730:0:55751:0:(file.c:7076:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000405:0x44:0x0](ffff97688b7baa08) 00000002:00010000:2.0:1787826426.837744:0:55751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x44:0x0]) in obj [0x200000405:0x44:0x0], intent: layout flags 0 00010000:00010000:2.0:1787826426.837812:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab188d lrc: 3/1,0 mode: --/CR res: [0x200000405:0x44:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.838020:0:55751:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab188d lrc: 3/1,0 mode: --/CR res: [0x200000405:0x44:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.838033:0:55751:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab188d lrc: 3/1,0 mode: --/CR res: [0x200000405:0x44:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826426.838057:0:55751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564011776:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826426.838161:0:55751:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35b40 going to sleep for 11 seconds 00000100:00100000:2.0:1787826426.842079:0:55751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564011776:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826426.842120:0:55751:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab188d lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x44:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83a72 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.842140:0:55751:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab188d lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x44:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a72 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826426.842156:0:55751:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff976889cf8780 x1874670564011776/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 376/480 e 0 to 0 dl 1787826442 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826426.842168:0:55751:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab188d lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x44:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a72 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787826426.842192:0:55751:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab188d lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x44:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a72 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.842213:0:55751:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/68 (0 0) 00000080:00200000:2.0:1787826426.842221:0:55751:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000405:0x44:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1787826426.842246:0:55751:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x44:0x0](ffff97688b7baa08) for lock 0x8e7c99ee62ab188d 00000080:00010000:2.0:1787826426.842256:0:55751:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab188d from it ffffb2d4422aba78 00010000:00010000:2.0:1787826426.842260:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab188d lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x44:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83a72 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.842271:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab188d lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x44:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a72 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.843014:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab188d lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x44:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a72 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.843033:0:55751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab188d lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x44:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a72 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1787826426.843044:0:55751:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x44:0x0](0000000073bfdfab) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab188d lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x44:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a72 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826426.843096:0:55751:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x44:0x0]: layout version change: 4294967294 -> 1 00010000:00010000:2.0:1787826426.843102:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab188d lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x44:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83a72 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.843111:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab188d lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x44:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83a72 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:2.0:1787826426.843138:0:55751:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787826426.843160:0:55751:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:2.0:1787826426.843189:0:55751:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1787826426.843195:0:55751:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff97688757cb30, osc lock ffff976883f7a270, flags 0x0 00010000:00010000:2.0:1787826426.843210:0:55751:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9768ad0ae800 type 11 mode 2 res 11811161089/271 (0 18446744073709551615) 00010000:00010000:2.0:1787826426.843256:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.843265:0:55751:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.843276:0:55751:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826426.843316:0:55751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564011904:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826426.843886:0:55751:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e359c0 going to sleep for 11 seconds 00000100:00100000:1.0:1787826426.846074:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826426.846410:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585294720 00000100:00100000:0.0:1787826426.846421:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585294720 00000100:00100000:0.0:1787826426.846533:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 624 00000100:00100000:0.0:1787826426.846545:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0163c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:14864:x1874670585294720:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826426.846569:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab186a lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b83a25 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826426.846674:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826426.846683:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0163c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:14864:x1874670585294720:12345-192.168.203.139@tcp:104: Request processed in 141us (624us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826426.846690:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 624 00010000:00010000:2.0:1787826426.846765:0:49796:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826426.846775:0:49796:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab186a lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b83a25 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.846783:0:49796:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515400 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:2.0:1787826426.846790:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab186a lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x22153cd396b83a25 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826426.846914:0:49796:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x44:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787826426.846930:0:49796:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x44:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1787826426.846994:0:49796:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab186a lrc: 2/0,0 mode: --/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b83a25 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.847009:0:49796:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826426.847032:0:49796:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab186a lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b83a25 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.847040:0:49796:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000d1835e52/0x8e7c99ee62ab186a lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b83a25 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.847060:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.847061:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.847063:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826426.847346:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564012032:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826426.860343:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:2.0:1787826426.860541:0:55751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564011904:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826426.860584:0:55751:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 4/0,1 mode: --/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.860601:0:55751:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 4/0,1 mode: --/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.860610:0:55751:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 4/0,1 mode: --/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826426.860695:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585294848 00000100:00100000:0.0:1787826426.860705:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585294848 00000100:00100000:0.0:1787826426.860813:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 625 00000100:00100000:0.0:1787826426.860897:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0161a40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:6503:x1874670585294848:12345-192.168.203.139@tcp:105: 00010000:00010000:0.0:1787826426.860921:0:42211:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 5/0,1 mode: --/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.860948:0:42211:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 5/0,1 mode: --/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.860963:0:42211:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 5/0,1 mode: --/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.860983:0:42211:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.860997:0:42211:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 6/0,1 mode: PW/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.861011:0:42211:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 6/0,1 mode: PW/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.861038:0:42211:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff976883514a00) 00000100:00100000:0.0:1787826426.861132:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0161a40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:6503:x1874670585294848:12345-192.168.203.139@tcp:105: Request processed in 307us (803us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826426.861143:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 625 00010000:00010000:2.0:1787826426.861212:0:55751:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.861223:0:55751:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.861242:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787826426.861257:0:55751:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 4/0,2 mode: PW/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.861272:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.861281:0:55751:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826426.861496:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826426.861517:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564012160:192.168.203.139@tcp:2:lfs.0 00000100:00100000:3.0:1787826426.868533:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564012160:192.168.203.139@tcp:2:lfs.0 00000100:00100000:3.0:1787826426.868557:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564012032:192.168.203.139@tcp:103:ldlm_bl.0 00000020:00010000:2.0:1787826426.875115:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976890fb74e8 (ffff976890d10128/0) at cl_lock_release():245 00000020:00010000:2.0:1787826426.875130:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976890fb74e8 (ffff976890d10128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787826426.875138:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757cb30 (ffff976890d10a28/1) at cl_lock_cancel():142 00010000:00010000:2.0:1787826426.875153:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.875184:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1787826426.875207:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976890fb74e8 (ffff976890d10128/0) at cl_lock_fini():72 00000020:00010000:2.0:1787826426.875210:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757cb30 (ffff976890d10128/1) at cl_lock_fini():72 00000080:00200000:2.0:1787826426.875240:0:55751:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x44:0x0] ignore/verify layout 0/0, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1787826426.875259:0:55751:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file . :VOLATILE:0000:4E0C50C4:fd=03:[0x200000405:0x44:0x0](ffff97688b7baa08) current size 0, valid attrs 100101f0, mode 0, uid 0, gid 0, new size 4294967296, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826426.000000000, rc 0 00000080:00200008:2.0:1787826426.875339:0:55751:0:(llite_lib.c:2427:ll_setattr_raw()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8) current size 0, valid attrs 101f0, mode 9768, uid 0, gid 0, new size 0, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826426.000000000 00010000:00010000:2.0:1787826426.875383:0:55751:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826426.875412:0:55751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1832 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b839ae expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826426.876074:0:55751:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 1/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826426.876085:0:55751:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826426.876088:0:55751:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514c00/0x8e7c99ee62ab1855 lrc: 0/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b83a09 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000020:00100000:2.0:1787826426.876117:0:55751:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787826426.876135:0:55751:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889cfa940 x1874670564012416/t0(0) o36->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826426.876164:0:55751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564012416:192.168.203.139@tcp:36:lfs.0 00000100:00100000:2.0:1787826426.876307:0:55751:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35c00 going to sleep for 11 seconds 00000100:00100000:2.0:1787826426.884521:0:55751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564012416:192.168.203.139@tcp:36:lfs.0 00000080:00200000:2.0:1787826426.884587:0:55751:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 1 restore needed 0 00020000:00200000:2.0:1787826426.884610:0:55751:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787826426.884629:0:55751:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:2.0:1787826426.884650:0:55751:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1787826426.884656:0:55751:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff97688757cb30, osc lock ffff976883f7a270, flags 0x0 00010000:00010000:2.0:1787826426.884675:0:55751:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976889533000 type 11 mode 2 res 10737419265/270 (0 18446744073709551615) 00010000:00010000:2.0:1787826426.884734:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.884747:0:55751:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.884759:0:55751:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 3/0,1 mode: --/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826426.884783:0:55751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8b40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564012544:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826426.884907:0:55751:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35c00 going to sleep for 11 seconds 00000100:00100000:1.0:1787826426.887745:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826426.887881:0:53061:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:0.0:1787826426.887907:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585294976 00000100:00100000:0.0:1787826426.887918:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585294976 00000100:00100000:0.0:1787826426.887987:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 626 00000100:00100000:0.0:1787826426.887999:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff97688a33e450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8436:x1874670585294976:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826426.888022:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826426.888119:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826426.888126:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff97688a33e450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8436:x1874670585294976:12345-192.168.203.139@tcp:104: Request processed in 130us (397us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826426.888138:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 626 00010000:00010000:2.0:1787826426.888414:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826426.888425:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826426.888434:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883515000 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:2.0:1787826426.888441:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1787826426.888464:0:42220:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787826426.888481:0:42220:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1787826426.888535:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 2/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826426.888545:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826426.888575:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826426.888584:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000006520b9ac/0x8e7c99ee62ab1886 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x22153cd396b83a56 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 0 00000100:00100000:3.0:1787826426.888601:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564012672:192.168.203.139@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1787826426.888604:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.888607:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.888610:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826426.891874:0:55751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8b40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564012544:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826426.891916:0:55751:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 4/0,1 mode: --/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b83a87 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.891933:0:55751:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 4/0,1 mode: --/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b83a87 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.891942:0:55751:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 4/0,1 mode: --/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b83a87 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826426.893424:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826426.893643:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585295104 00000100:00100000:0.0:1787826426.893654:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585295104 00000100:00100000:0.0:1787826426.893728:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 627 00000100:00100000:0.0:1787826426.893741:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0161680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:10340:x1874670585295104:12345-192.168.203.139@tcp:105: 00010000:00010000:0.0:1787826426.893758:0:42211:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 5/0,1 mode: --/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b83a87 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.893773:0:42211:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 5/0,1 mode: --/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b83a87 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.893784:0:42211:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 5/0,1 mode: --/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b83a87 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.893796:0:42211:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x22153cd396b83a87 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.893807:0:42211:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 6/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x22153cd396b83a87 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.893817:0:42211:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 6/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x22153cd396b83a87 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826426.893893:0:42211:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff976883514c00) 00000100:00100000:0.0:1787826426.893975:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0161680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:10340:x1874670585295104:12345-192.168.203.139@tcp:105: Request processed in 236us (561us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826426.893984:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 627 00010000:00010000:2.0:1787826426.894106:0:55751:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b83a87 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.894119:0:55751:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x22153cd396b83a87 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.894139:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x22153cd396b83a87 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787826426.894155:0:55751:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x22153cd396b83a87 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.894181:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b83a87 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.894191:0:55751:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b83a87 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826426.894792:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826426.894818:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564012800:192.168.203.139@tcp:2:lfs.0 00000100:00100000:3.0:1787826426.899155:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564012672:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826426.900188:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564012800:192.168.203.139@tcp:2:lfs.0 00000020:00010000:2.0:1787826426.900243:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976890fb74e8 (ffff976890d10128/0) at cl_lock_release():245 00000020:00010000:2.0:1787826426.900250:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976890fb74e8 (ffff976890d10128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787826426.900257:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757cb30 (ffff976890d10a28/1) at cl_lock_cancel():142 00010000:00010000:2.0:1787826426.900270:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b83a87 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.900289:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b83a87 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1787826426.900311:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976890fb74e8 (ffff976890d10128/0) at cl_lock_fini():72 00000020:00010000:2.0:1787826426.900314:0:55751:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757cb30 (ffff976890d10128/1) at cl_lock_fini():72 00000080:00200000:2.0:1787826426.900338:0:55751:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/0, layout version 39 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1787826426.900475:0:55751:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8) current size 0, valid attrs 100101f0, mode 9768, uid 0, gid 0, new size 0, atime 1787826412.000000000, mtime 1787826412.000000000, ctime 1787826426.000000000, rc 0 00000080:00200080:2.0:1787826426.900512:0:55751:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=ef1570 00010000:00010000:2.0:1787826426.900660:0:55751:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1871 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0x22153cd396b83a33 expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.900682:0:55751:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab1871 lrc: 0/0,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x85809400000000 nid: local remote: 0x22153cd396b83a33 expref: -99 pid: 55751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826426.900707:0:55751:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889efb0c0 x1874670564010240/t8589935275(8589935275) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826442 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787826426.900720:0:55751:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787826426.900726:0:55751:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889cf9e00 x1874670564013056/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 504/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787826426.900754:0:55751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564013056:192.168.203.139@tcp:35:lfs.0 00000100:00100000:2.0:1787826426.901009:0:55751:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35c00 going to sleep for 11 seconds 00000100:00100000:1.0:1787826426.903276:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826426.903321:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585295232 00000100:00100000:1.0:1787826426.903335:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585295232 00000100:00100000:1.0:1787826426.903422:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 628 00000100:00100000:1.0:1787826426.903435:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0161e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585295232:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826426.903462:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1832 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b839ae expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1787826426.903616:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826426.903624:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0161e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585295232:12345-192.168.203.139@tcp:104: Request processed in 192us (363us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826426.903632:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 628 00000100:00100000:1.0:1787826426.903691:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826426.903892:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585295360 00000100:00100000:1.0:1787826426.903897:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585295360 00000100:00100000:1.0:1787826426.903926:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 629 00000100:00100000:1.0:1787826426.903933:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0160b40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585295360:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826426.903944:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83961 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.906063:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:2.0:1787826426.906082:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1832 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b839ae expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826426.906097:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976888fb9400 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:2.0:1787826426.906104:0:42221:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1832 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b839ae expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826426.906114:0:42221:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1832 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b839ae expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826426.906124:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1832 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 3 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b839ae expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826426.906163:0:42221:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83961 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000002:00100000:2.0:1787826426.906255:0:42221:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff97688cb5d2c0 x1874670564007936/t8589935270(8589935270) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826442 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787826426.906271:0:42221:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 2 is allocated opc 35, max 7 00000100:00100000:2.0:1787826426.906278:0:42221:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889cfa580 x1874670564013312/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787826426.906310:0:42221:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:42221:1874670564013312:192.168.203.139@tcp:35:ldlm_bl.0 00000100:00100000:2.0:1787826426.906733:0:42221:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e359c0 going to sleep for 11 seconds 00010000:00010000:1.0:1787826426.906837:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826426.906851:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0160b40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585295360:12345-192.168.203.139@tcp:104: Request processed in 2915us (3157us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826426.906864:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 629 00010000:00010000:2.0:1787826426.906957:0:49796:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826426.906965:0:49796:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83961 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.906975:0:49796:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514800 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:2.0:1787826426.906979:0:49796:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83961 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.906987:0:49796:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83961 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.906995:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b83961 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787826426.907017:0:49796:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x1d:0x0]: losing layout lock 00010000:00010000:2.0:1787826426.907063:0:49796:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83961 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.907070:0:49796:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826426.907102:0:49796:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83961 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.907111:0:49796:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab181d lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83961 expref: -99 pid: 55743 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787826426.907129:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.907131:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.907133:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826426.907202:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564013440:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826426.914694:0:42221:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:42221:1874670564013312:192.168.203.139@tcp:35:ldlm_bl.0 00000002:00100000:2.0:1787826426.914743:0:42221:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff97688cb5d2c0 x1874670564007936/t8589935270(8589935270) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826442 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1787826426.914955:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1832 lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b839ae expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826426.914966:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826426.914999:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1832 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b839ae expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826426.915008:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9400/0x8e7c99ee62ab1832 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0xd rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b839ae expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787826426.915034:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.915036:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.915038:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826426.915204:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826426.915229:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564013568:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826426.918790:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826426.918933:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585295488 00000100:00100000:1.0:1787826426.918948:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585295488 00000100:00100000:1.0:1787826426.919018:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 630 00000100:00100000:1.0:1787826426.919029:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aee580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585295488:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826426.919052:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab188d lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x44:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83a72 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826426.919151:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826426.919158:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aee580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585295488:12345-192.168.203.139@tcp:104: Request processed in 131us (385us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826426.919166:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 630 00010000:00010000:2.0:1787826426.919208:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826426.919218:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab188d lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x44:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83a72 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.919230:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514000 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:2.0:1787826426.919237:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab188d lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x44:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83a72 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.919248:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab188d lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x44:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83a72 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.919258:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab188d lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x44:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b83a72 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826426.919290:0:42220:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x44:0x0]: losing layout lock 00010000:00010000:2.0:1787826426.919345:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab188d lrc: 2/0,0 mode: --/CR res: [0x200000405:0x44:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83a72 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.919353:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826426.919375:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab188d lrc: 1/0,0 mode: --/CR res: [0x200000405:0x44:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83a72 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.919384:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab188d lrc: 0/0,0 mode: --/CR res: [0x200000405:0x44:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83a72 expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.919407:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.919409:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826426.919410:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826426.919711:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564013696:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826426.919871:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564013568:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826426.919886:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564013440:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826426.926914:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564013696:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826426.928268:0:55751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564013056:192.168.203.139@tcp:35:lfs.0 00000002:00100000:2.0:1787826426.928321:0:55751:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889efb0c0 x1874670564010240/t8589935275(8589935275) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826442 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1787826426.928380:0:55751:0:(file.c:7076:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000002:00010000:2.0:1787826426.928395:0:55751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: layout flags 0 00010000:00010000:2.0:1787826426.928460:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.928484:0:55751:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.928497:0:55751:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55751 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826426.928522:0:55751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564013824:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826426.928643:0:55751:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35c00 going to sleep for 11 seconds 00000100:00100000:2.0:1787826426.930667:0:55751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564013824:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826426.930703:0:55751:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83a9c expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.930721:0:55751:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a9c expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826426.930737:0:55751:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff976889cf9e00 x1874670564013824/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 376/584 e 0 to 0 dl 1787826442 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826426.930749:0:55751:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a9c expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787826426.930772:0:55751:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a9c expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.930786:0:55751:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:2.0:1787826426.930794:0:55751:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1787826426.930812:0:55751:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab18a2 00000080:00010000:2.0:1787826426.930822:0:55751:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab18a2 from it ffffb2d4422abbc8 00010000:00010000:2.0:1787826426.930913:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83a9c expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.930925:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a9c expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.930958:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a9c expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.930970:0:55751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a9c expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1787826426.930980:0:55751:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a9c expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.931027:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83a9c expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826426.931036:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a9c expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826426.931054:0:55751:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787826426.931068:0:55751:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787826426.931104:0:55751:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:2.0:1787826426.931117:0:55751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a9c expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.931125:0:55751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a9c expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1787826426.931131:0:55751:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83a9c expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826426.931146:0:55751:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787826426.931151:0:55751:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787826426.931158:0:55751:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000008:00010000:2.0:1787826426.931183:0:55751:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 0 [cma]time: 1787826426 1787826412 1787826412 by oinfo size 0 blocks 0 [cma]time 1787826426 1787826412 1787826412 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 1/0,0 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b83a87 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826426.931290:0:55751:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 4294967294 -> 40 00010000:00010000:2.0:1787826426.931297:0:55751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83a9c expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826426.931304:0:55751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83a9c expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:2.0:1787826426.931341:0:55751:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 1154002 00000002:00100000:2.0:1787826426.932109:0:55751:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889efa940 x1874670564008320/t8589935271(8589935271) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826442 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787826426.932126:0:55751:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787826426.932133:0:55751:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889cf9e00 x1874670564014080/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787826426.932163:0:55751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564014080:192.168.203.139@tcp:35:lfs.0 00000100:00100000:2.0:1787826426.932307:0:55751:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35900 going to sleep for 11 seconds 00000100:00100000:2.0:1787826426.935524:0:55751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564014080:192.168.203.139@tcp:35:lfs.0 00000002:00100000:2.0:1787826426.935739:0:55751:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889efa940 x1874670564008320/t8589935271(8589935271) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 656/744 e 0 to 0 dl 1787826442 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1787826426.935921:0:55751:0:(file.c:479:ll_file_release()) START file . :VOLATILE:0000:4E0C50C4:fd=03:[0x200000405:0x44:0x0](ffff97688b7baa08), flags 554001 00000002:00100000:2.0:1787826426.935985:0:55751:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889efb480 x1874670564008704/t8589935272(8589935272) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826442 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787826426.935996:0:55751:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787826426.936002:0:55751:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889cf9e00 x1874670564014336/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787826426.936030:0:55751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564014336:192.168.203.139@tcp:35:lfs.0 00000100:00100000:2.0:1787826426.936198:0:55751:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35900 going to sleep for 11 seconds 00000100:00100000:2.0:1787826426.939924:0:55751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55751:1874670564014336:192.168.203.139@tcp:35:lfs.0 00000002:00100000:2.0:1787826426.939971:0:55751:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889efb480 x1874670564008704/t8589935272(8589935272) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826442 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1787826426.959941:0:55753:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:2.0:1787826426.959985:0:55753:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826426.960010:0:55753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826426.960027:0:55753:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826426.960037:0:55753:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826426.960058:0:55753:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:2.0:1787826426.960063:0:55753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:2.0:1787826426.960073:0:55753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d442253b58 00010000:00010000:2.0:1787826426.960077:0:55753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826426.960087:0:55753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826426.960467:0:55753:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:2.0:1787826426.960477:0:55753:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=24 00000080:00200000:2.0:1787826426.960483:0:55753:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787826426.960495:0:55753:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), flags=8 00000080:00200000:2.0:1787826426.960500:0:55753:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=getattr 00000002:00010000:2.0:1787826426.960518:0:55753:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f60b.sanity-flr,[0x0:0x0:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787826426.960586:0:55753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55753 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.960594:0:55753:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55753 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826426.960605:0:55753:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55753 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826426.960637:0:55753:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55753:1874670564014464:192.168.203.139@tcp:101:getfattr.0 00000100:00100000:2.0:1787826426.960765:0:55753:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35900 going to sleep for 11 seconds 00000100:00100000:3.0:1787826426.964306:0:55753:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55753:1874670564014464:192.168.203.139@tcp:101:getfattr.0 00010000:00010000:3.0:1787826426.964339:0:55753:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 4/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.964381:0:55753:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.964390:0:55753:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.964404:0:55753:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.964415:0:55753:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.964430:0:55753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.964439:0:55753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.964447:0:55753:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826426.964460:0:55753:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff976889cf8000 x1874670564014464/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 584/824 e 0 to 0 dl 1787826442 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'getfattr.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826426.964472:0:55753:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826426.964492:0:55753:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.964503:0:55753:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826426.964511:0:55753:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1787826426.964577:0:55753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x1d:0x0], updating i_size 0 00000080:00200000:3.0:1787826426.964581:0:55753:0:(namei.c:147:ll_iget()) got inode: [0x200000405:0x1d:0x0](ffff9768901a80c8): rc = 0 00000080:00200000:3.0:1787826426.964595:0:55753:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 40 -> 40 00000080:00010000:3.0:1787826426.964601:0:55753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab18a9 00000080:00010000:3.0:1787826426.964614:0:55753:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:3.0:1787826426.964617:0:55753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab18a9 from it ffffb2d442253b20 00010000:00010000:3.0:1787826426.964621:0:55753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826426.964631:0:55753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1787826426.964672:0:55753:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826426.964679:0:55753:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:3.0:1787826426.964706:0:55753:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826426.964716:0:55753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826426.964727:0:55753:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826426.964740:0:55753:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826426.964746:0:55753:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:3.0:1787826426.964750:0:55753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab18a9 00000080:00010000:3.0:1787826426.964753:0:55753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab18a9 from it ffffb2d442253cb8 00010000:00010000:3.0:1787826426.964756:0:55753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826426.964764:0:55753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826426.964810:0:55753:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826426.964817:0:55753:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826426.964851:0:55753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826426.964860:0:55753:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826426.964867:0:55753:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826426.964870:0:55753:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826426.964873:0:55753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826426.964877:0:55753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d442253bf8 00010000:00010000:3.0:1787826426.964880:0:55753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826426.964887:0:55753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826426.964896:0:55753:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826426.964904:0:55753:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:3.0:1787826426.964909:0:55753:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787826426.964912:0:55753:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=9 00000080:00200000:3.0:1787826426.964920:0:55753:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:3.0:1787826426.964928:0:55753:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200000:3.0:1787826426.964947:0:55753:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826426.964953:0:55753:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826426.964960:0:55753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826426.964968:0:55753:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826426.964974:0:55753:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826426.964977:0:55753:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826426.964979:0:55753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826426.964983:0:55753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d442253bf8 00010000:00010000:3.0:1787826426.964986:0:55753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826426.964993:0:55753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826426.965001:0:55753:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826426.965005:0:55753:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:3.0:1787826426.965009:0:55753:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787826426.965012:0:55753:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=9 00000080:00200000:3.0:1787826426.965019:0:55753:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:3.0:1787826426.965023:0:55753:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200000:3.0:1787826427.016606:0:55757:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826427.016644:0:55757:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826427.016668:0:55757:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826427.016687:0:55757:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826427.016707:0:55757:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826427.016715:0:55757:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826427.016719:0:55757:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826427.016728:0:55757:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4422c3bf8 00010000:00010000:3.0:1787826427.016733:0:55757:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826427.016742:0:55757:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826427.016761:0:55757:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826427.016771:0:55757:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=27 00000080:00200000:3.0:1787826427.016777:0:55757:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787826427.016782:0:55757:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=267 00000080:00200000:3.0:1787826427.016817:0:55757:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:3.0:1787826427.017024:0:55757:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826427.017033:0:55757:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826427.017042:0:55757:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826427.017049:0:55757:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826427.017053:0:55757:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:3.0:1787826427.017055:0:55757:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:3.0:1787826427.017059:0:55757:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4422c3bf8 00010000:00010000:3.0:1787826427.017062:0:55757:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826427.017069:0:55757:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787826427.017078:0:55757:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:3.0:1787826427.017082:0:55757:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=267 00000080:00200000:3.0:1787826427.017087:0:55757:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 44 00000080:00200008:3.0:1787826427.017094:0:55757:0:(file.c:1090:ll_file_open()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00000080:00200080:3.0:1787826427.017129:0:55757:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8) cmd=c0086915 arg=1cf33d0 00000100:00100000:3.0:1787826427.017235:0:55757:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55757:1874670564014592:192.168.203.139@tcp:34:lfs.0 00000100:00100000:3.0:1787826427.017422:0:55757:0:(client.c:2715:ptlrpc_set_wait()) set ffff976886b25900 going to sleep for 11 seconds 00000100:00100000:3.0:1787826427.020752:0:55757:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55757:1874670564014592:192.168.203.139@tcp:34:lfs.0 00000080:00200008:3.0:1787826427.021099:0:55757:0:(file.c:479:ll_file_release()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00010000:00010000:3.0:1787826427.021142:0:55757:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9600/0x8e7c99ee62ab0c1c lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b818c9 expref: -99 pid: 55483 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000080:00200000:0.0:1787826427.045704:0:55759:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826427.045752:0:55759:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826427.045779:0:55759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826427.045799:0:55759:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826427.045809:0:55759:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826427.045820:0:55759:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826427.045856:0:55759:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826427.045867:0:55759:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4422c3bf8 00010000:00010000:0.0:1787826427.045872:0:55759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826427.045883:0:55759:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826427.045905:0:55759:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826427.045914:0:55759:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:0.0:1787826427.045921:0:55759:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826427.045926:0:55759:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:0.0:1787826427.045931:0:55759:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 44 00000080:00200008:0.0:1787826427.045939:0:55759:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154400 00000002:00010000:0.0:1787826427.045961:0:55759:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114401 00010000:00010000:0.0:1787826427.046030:0:55759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18b0 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 55759 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.046039:0:55759:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18b0 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55759 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826427.046053:0:55759:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826427.046070:0:55759:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aef0c0 x1874670564014848/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826427.046082:0:55759:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18b0 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 55759 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826427.046110:0:55759:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564014848:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826427.047958:0:55759:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e240 going to sleep for 11 seconds 00000100:00100000:3.0:1787826427.060305:0:55759:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564014848:192.168.203.139@tcp:101:lfs.0 00010000:00010000:3.0:1787826427.060459:0:55759:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18b0 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83ae2 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826427.060482:0:55759:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18b0 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83ae2 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826427.060495:0:55759:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18b0 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83ae2 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826427.060515:0:55759:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff976882aef0c0 x1874670564014848/t8589935284(8589935284) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826443 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826427.060529:0:55759:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aef0c0 x1874670564014848/t8589935284(8589935284) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826443 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826427.060543:0:55759:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18b0 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83ae2 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826427.060561:0:55759:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18b0 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83ae2 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826427.060575:0:55759:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:3.0:1787826427.060582:0:55759:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826427.060640:0:55759:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x1d:0x0], updating i_size 0 00000080:00200000:3.0:1787826427.060656:0:55759:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 40 -> 40 00000080:00010000:3.0:1787826427.060661:0:55759:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab18b0 00000080:00010000:3.0:1787826427.060670:0:55759:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab18b0 from it ffffb2d4422c3bf0 00010000:00010000:3.0:1787826427.060675:0:55759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18b0 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83ae2 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826427.060684:0:55759:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18b0 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83ae2 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000002:00100000:3.0:1787826427.060797:0:55759:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aef0c0 x1874670564014848/t8589935284(8589935284) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826443 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826427.061282:0:55759:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:0.0:1787826427.061293:0:55759:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:0.0:1787826427.061329:0:55759:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826427.061337:0:55759:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:0.0:1787826427.061487:0:55759:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826427.061503:0:55759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826427.061516:0:55759:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826427.061523:0:55759:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826427.061530:0:55759:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:0.0:1787826427.061534:0:55759:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab18a9 00000080:00010000:0.0:1787826427.061541:0:55759:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab18a9 from it ffffb2d4422c3cd8 00010000:00010000:0.0:1787826427.061544:0:55759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826427.061552:0:55759:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:0.0:1787826427.061606:0:55759:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154000 00010000:00010000:0.0:1787826427.061621:0:55759:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18b0 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83ae2 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826427.062021:0:55759:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826427.062034:0:55759:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826427.062045:0:55759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826427.062054:0:55759:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826427.062061:0:55759:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826427.062065:0:55759:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826427.062067:0:55759:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826427.062072:0:55759:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4422c3bf8 00010000:00010000:0.0:1787826427.062075:0:55759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826427.062084:0:55759:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826427.062178:0:55759:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826427.062187:0:55759:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:0.0:1787826427.062193:0:55759:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826427.062197:0:55759:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:0.0:1787826427.062202:0:55759:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 44 00000080:00200008:0.0:1787826427.062210:0:55759:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154400 00000080:00200000:0.0:1787826427.062273:0:55759:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:0.0:1787826427.062279:0:55759:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200008:0.0:1787826427.062301:0:55759:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826427.062305:0:55759:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:0.0:1787826427.062316:0:55759:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826427.062324:0:55759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826427.062333:0:55759:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826427.062340:0:55759:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826427.062473:0:55759:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:0.0:1787826427.062476:0:55759:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab18a9 00000080:00010000:0.0:1787826427.062481:0:55759:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab18a9 from it ffffb2d4422c3cd8 00010000:00010000:0.0:1787826427.062485:0:55759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826427.062493:0:55759:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200008:0.0:1787826427.062523:0:55759:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154000 00010000:00010000:0.0:1787826427.062534:0:55759:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18b0 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83ae2 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826427.062561:0:55759:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826427.062569:0:55759:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826427.062574:0:55759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826427.062581:0:55759:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826427.062587:0:55759:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826427.062591:0:55759:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826427.062594:0:55759:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826427.062598:0:55759:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4422c3bf8 00010000:00010000:0.0:1787826427.062601:0:55759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826427.062608:0:55759:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826427.062616:0:55759:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826427.062620:0:55759:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:0.0:1787826427.062623:0:55759:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826427.062626:0:55759:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=265 00000080:00200000:0.0:1787826427.062629:0:55759:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 46 00000080:00200008:0.0:1787826427.062633:0:55759:0:(file.c:1090:ll_file_open()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154402 00000002:00010000:0.0:1787826427.062643:0:55759:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 0104240114403 00010000:00010000:0.0:1787826427.062702:0:55759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925da00/0x8e7c99ee62ab18b7 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 55759 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.062792:0:55759:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925da00/0x8e7c99ee62ab18b7 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55759 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826427.062808:0:55759:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826427.062822:0:55759:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aecf00 x1874670564015104/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826427.063003:0:55759:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925da00/0x8e7c99ee62ab18b7 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 55759 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826427.063239:0:55759:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aecf00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564015104:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826427.063547:0:55759:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35c00 going to sleep for 11 seconds 00000100:00100000:1.0:1787826427.066429:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826427.066530:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585296000 00000100:00100000:0.0:1787826427.066543:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585296000 00000100:00100000:0.0:1787826427.066612:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 631 00000100:00100000:0.0:1787826427.066623:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0163c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:8247:x1874670585296000:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826427.066646:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826427.066735:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826427.066742:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0163c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:8247:x1874670585296000:12345-192.168.203.139@tcp:104: Request processed in 121us (328us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826427.066750:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 631 00010000:00010000:2.0:1787826427.066811:0:49796:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826427.066820:0:49796:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826427.066871:0:49796:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514800 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:2.0:1787826427.066877:0:49796:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826427.066887:0:49796:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826427.066896:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826427.066929:0:49796:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18b0 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83ae2 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826427.066981:0:49796:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 2/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826427.066989:0:49796:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826427.067021:0:49796:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 1/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826427.067035:0:49796:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18a9 lrc: 0/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83abf expref: -99 pid: 55753 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826427.067054:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826427.067055:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826427.067057:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826427.067400:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564015232:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826427.071303:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564015232:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826427.074157:0:55759:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aecf00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564015104:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826427.074212:0:55759:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode CW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925da00/0x8e7c99ee62ab18b7 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83af7 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.074227:0:55759:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925da00/0x8e7c99ee62ab18b7 lrc: 4/1,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83af7 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.074241:0:55759:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925da00/0x8e7c99ee62ab18b7 lrc: 4/1,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83af7 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.074254:0:55759:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925da00/0x8e7c99ee62ab18b7 lrc: 4/1,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83af7 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.074271:0:55759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925da00/0x8e7c99ee62ab18b7 lrc: 5/1,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83af7 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.074280:0:55759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925da00/0x8e7c99ee62ab18b7 lrc: 5/1,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83af7 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.074288:0:55759:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925da00/0x8e7c99ee62ab18b7 lrc: 4/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83af7 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826427.074305:0:55759:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff976882aecf00 x1874670564015104/t8589935285(8589935285) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826443 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787826427.074319:0:55759:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aecf00 x1874670564015104/t8589935285(8589935285) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826443 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826427.074334:0:55759:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925da00/0x8e7c99ee62ab18b7 lrc: 3/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83af7 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.074350:0:55759:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 8 res 8589935621/29 (0 0) 00000080:00200000:2.0:1787826427.074358:0:55759:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787826427.074422:0:55759:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x1d:0x0], updating i_size 0 00000080:00010000:2.0:1787826427.074429:0:55759:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab18b7 00000080:00010000:2.0:1787826427.074437:0:55759:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab18b7 from it ffffb2d4422c3bf0 00010000:00010000:2.0:1787826427.074441:0:55759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925da00/0x8e7c99ee62ab18b7 lrc: 3/0,1 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83af7 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.074451:0:55759:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925da00/0x8e7c99ee62ab18b7 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83af7 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:2.0:1787826427.074485:0:55759:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aecf00 x1874670564015104/t8589935285(8589935285) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826443 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:2.0:1787826427.074651:0:55759:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=800466af arg=7fff9a2130ec 00000080:00200080:2.0:1787826427.074672:0:55759:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=80086601 arg=7fff9a213088 00000100:00100000:2.0:1787826427.074730:0:55759:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564015360:192.168.203.139@tcp:33:lfs.0 00000100:00100000:2.0:1787826427.074867:0:55759:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826427.080585:0:55759:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564015360:192.168.203.139@tcp:33:lfs.0 00000080:00200080:0.0:1787826427.080646:0:55759:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=7fff9a213074 00000002:00010000:0.0:1787826427.080683:0:55759:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 01004040000002 00010000:00010000:0.0:1787826427.080706:0:55759:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925da00/0x8e7c99ee62ab18b7 lrc: 2/0,0 mode: CW/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x22153cd396b83af7 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826427.080759:0:55759:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18b0 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b83ae2 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826427.080778:0:55759:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83a9c expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000002:00100000:0.0:1787826427.080850:0:55759:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976882aef0c0 x1874670564014848/t8589935284(8589935284) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826443 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826427.080862:0:55759:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826427.080869:0:55759:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aeda40 x1874670564015616/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826427.080895:0:55759:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeda40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564015616:192.168.203.139@tcp:35:lfs.0 00000100:00100000:0.0:1787826427.081004:0:55759:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e9c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826427.084281:0:55759:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeda40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564015616:192.168.203.139@tcp:35:lfs.0 00000002:00100000:0.0:1787826427.084328:0:55759:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976882aef0c0 x1874670564014848/t8589935284(8589935284) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826443 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1787826427.084451:0:55759:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925da00/0x8e7c99ee62ab18b7 lrc: 1/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b83af7 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826427.084461:0:55759:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18b0 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b83ae2 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826427.084472:0:55759:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787826427.084475:0:55759:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925da00/0x8e7c99ee62ab18b7 lrc: 0/0,0 mode: --/CW res: [0x200000405:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b83af7 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826427.084488:0:55759:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18b0 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b83ae2 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826427.084514:0:55759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18be lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 55759 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.084522:0:55759:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x81000000001000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18be lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 55759 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826427.084533:0:55759:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826427.084539:0:55759:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aeda40 x1874670564015872/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 976/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826427.084549:0:55759:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18be lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x81000000000000 pid: 55759 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826427.084573:0:55759:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeda40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564015872:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826427.085001:0:55759:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e9c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826427.090849:0:55759:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeda40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564015872:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826427.090887:0:55759:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18be lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83b0c expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.090903:0:55759:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18be lrc: 4/1,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83b0c expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.090919:0:55759:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18be lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83b0c expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.090932:0:55759:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18be lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83b0c expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.090948:0:55759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18be lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83b0c expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.090957:0:55759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18be lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83b0c expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.090966:0:55759:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18be lrc: 4/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83b0c expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826427.090979:0:55759:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x600002b, status=0 req@ffff976882aeda40 x1874670564015872/t8589935287(8589935287) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 976/848 e 0 to 0 dl 1787826443 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826427.090994:0:55759:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aeda40 x1874670564015872/t8589935287(8589935287) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 976/848 e 0 to 0 dl 1787826443 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826427.091013:0:55759:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18be lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83b0c expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.091030:0:55759:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 2 res 8589935621/29 (0 0) 00000080:00200000:0.0:1787826427.091037:0:55759:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000002:00100000:0.0:1787826427.091064:0:55759:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aeda40 x1874670564015872/t8589935287(8589935287) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 768/848 e 0 to 0 dl 1787826443 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787826427.091073:0:55759:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab18be 00000080:00010000:0.0:1787826427.091080:0:55759:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab18be from it ffffb2d4422c3cc0 00010000:00010000:0.0:1787826427.091083:0:55759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18be lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83b0c expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.091459:0:55759:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18be lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83b0c expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200080:0.0:1787826427.091852:0:55759:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=c282e0 00010000:00010000:0.0:1787826427.091925:0:55759:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18be lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0x22153cd396b83b0c expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.091944:0:55759:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18be lrc: 0/0,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x85809400000000 nid: local remote: 0x22153cd396b83b0c expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826427.091966:0:55759:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976882aeda40 x1874670564015872/t8589935287(8589935287) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 768/848 e 0 to 0 dl 1787826443 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826427.091979:0:55759:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826427.091985:0:55759:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aef0c0 x1874670564016128/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 504/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826427.092011:0:55759:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564016128:192.168.203.139@tcp:35:lfs.0 00000100:00100000:0.0:1787826427.092971:0:55759:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1787826427.096963:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826427.097046:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585296128 00000100:00100000:0.0:1787826427.097058:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585296128 00000100:00100000:0.0:1787826427.097221:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 632 00000100:00100000:0.0:1787826427.097233:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0161a40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585296128:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826427.097258:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83a9c expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826427.097357:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826427.097364:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0161a40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:14257:x1874670585296128:12345-192.168.203.139@tcp:104: Request processed in 135us (662us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826427.097372:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 632 00010000:00010000:2.0:1787826427.097420:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826427.097429:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83a9c expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826427.097439:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514000 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:2.0:1787826427.097445:0:42221:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83a9c expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826427.097454:0:42221:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83a9c expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826427.097463:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b83a9c expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1787826427.097489:0:42221:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x1d:0x0]: losing layout lock 00010000:00010000:2.0:1787826427.097539:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83a9c expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826427.097545:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826427.097574:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83a9c expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826427.097582:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514000/0x8e7c99ee62ab18a2 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83a9c expref: -99 pid: 55751 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787826427.097605:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826427.097607:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826427.097609:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826427.097617:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1787826427.097633:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564016256:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826427.102240:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564016256:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826427.102321:0:55759:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564016128:192.168.203.139@tcp:35:lfs.0 00000080:00020000:0.0:1787826427.102337:0:55759:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9768877a4800: inode [0x200000405:0x1d:0x0] mdc close failed: rc = -22 00000002:00100000:0.0:1787826427.102477:0:55759:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976882aeda40 x1874670564015872/t8589935287(8589935287) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 768/848 e 0 to 0 dl 1787826443 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787826427.102517:0:55759:0:(file.c:7076:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000002:00010000:0.0:1787826427.102530:0:55759:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: layout flags 0 00010000:00010000:0.0:1787826427.102673:0:55759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18c5 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55759 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.102684:0:55759:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18c5 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55759 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.102695:0:55759:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18c5 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55759 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826427.102783:0:55759:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeda40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564016384:192.168.203.139@tcp:101:lfs.0 00000100:00100000:1.0:1787826427.103089:0:55759:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e9c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787826427.106818:0:55759:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeda40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564016384:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826427.106926:0:55759:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18c5 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83b21 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.106940:0:55759:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18c5 lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b21 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826427.106954:0:55759:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff976882aeda40 x1874670564016384/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 376/584 e 0 to 0 dl 1787826443 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826427.106963:0:55759:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18c5 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b21 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787826427.106973:0:55759:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18c5 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b21 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.106984:0:55759:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:2.0:1787826427.106990:0:55759:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1787826427.107007:0:55759:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab18c5 00000080:00010000:2.0:1787826427.107014:0:55759:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab18c5 from it ffffb2d4422c3bc8 00010000:00010000:2.0:1787826427.107017:0:55759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18c5 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83b21 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.107025:0:55759:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18c5 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b21 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826427.107055:0:55759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18c5 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b21 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826427.107064:0:55759:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18c5 lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b21 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1787826427.107070:0:55759:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18c5 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b21 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826427.107166:0:55759:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 4294967294 -> 40 00010000:00010000:2.0:1787826427.107180:0:55759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18c5 lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83b21 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826427.107189:0:55759:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18c5 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83b21 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826427.107887:0:55759:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:2.0:1787826427.107910:0:55759:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826427.107923:0:55759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826427.107931:0:55759:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826427.107935:0:55759:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826427.107941:0:55759:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:2.0:1787826427.107944:0:55759:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:2.0:1787826427.107948:0:55759:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4422c3bf8 00010000:00010000:2.0:1787826427.107950:0:55759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826427.107955:0:55759:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826427.107962:0:55759:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:2.0:1787826427.107968:0:55759:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=24 00000080:00200000:2.0:1787826427.107973:0:55759:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787826427.107986:0:55759:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 3 00000080:00200000:2.0:1787826427.107990:0:55759:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=. :VOLATILE:0000:734BD555:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), file ffff97688a0c2780, open_flags 2f9c2, mode 8180 opened 0 00000080:00200000:2.0:1787826427.107996:0:55759:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=. :VOLATILE:0000:734BD555:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=open|creat 00000002:00010000:2.0:1787826427.108021:0:55759:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: . :VOLATILE:0000:734BD555:fd=03,[0x200000405:0x45:0x0]) in obj [0x200000bd0:0x4:0x0], intent: open|creat flags 0574703 00010000:00010000:2.0:1787826427.108160:0:55759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18cc lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55759 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.108179:0:55759:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18cc lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55759 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1787826427.108193:0:55759:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1787826427.108204:0:55759:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889cfbc00 x1874670564016640/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/66640 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1787826427.108214:0:55759:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18cc lrc: 3/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55759 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826427.108243:0:55759:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564016640:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826427.108711:0:55759:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35b40 going to sleep for 11 seconds 00000100:00100000:2.0:1787826427.112064:0:55759:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564016640:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826427.112099:0:55759:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18cc lrc: 4/0,1 mode: --/CW res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b2f expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.112136:0:55759:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18cc lrc: 4/0,1 mode: --/PR res: [0x200000405:0x45:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b2f expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.112145:0:55759:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18cc lrc: 4/0,1 mode: --/PR res: [0x200000405:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b2f expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.112159:0:55759:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18cc lrc: 4/0,1 mode: PR/PR res: [0x200000405:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83b2f expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.112281:0:55759:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18cc lrc: 4/0,1 mode: PR/PR res: [0x200000405:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b2f expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.112300:0:55759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18cc lrc: 5/1,1 mode: PR/PR res: [0x200000405:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83b2f expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.112309:0:55759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18cc lrc: 5/1,1 mode: PR/PR res: [0x200000405:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83b2f expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.112318:0:55759:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18cc lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83b2f expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826427.112332:0:55759:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff976889cfbc00 x1874670564016640/t8589935289(8589935289) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826443 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787826427.112446:0:55759:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889cfbc00 x1874670564016640/t8589935289(8589935289) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826443 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826427.112463:0:55759:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18cc lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b2f expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.112475:0:55759:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/69 (0 0) 00000080:00200000:2.0:1787826427.112482:0:55759:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd0:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787826427.112497:0:55759:0:(namei.c:1628:ll_create_it()) VFS Op:name=. :VOLATILE:0000:734BD555:fd=03, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=open|creat 00000080:00200000:2.0:1787826427.112535:0:55759:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000405:0x45:0x0](ffff97688b7bb248) 00000080:00200000:2.0:1787826427.112544:0:55759:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000405:0x45:0x0], updating i_size 0 00000080:00200000:2.0:1787826427.112586:0:55759:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x45:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1787826427.112591:0:55759:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000405:0x45:0x0](ffff97688b7bb248) 00000080:00010000:2.0:1787826427.112595:0:55759:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x45:0x0](ffff97688b7bb248) for lock 0x8e7c99ee62ab18cc 00000080:00010000:2.0:1787826427.112607:0:55759:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x45:0x0](ffff97688b7bb248) 00000080:00200008:2.0:1787826427.112625:0:55759:0:(file.c:1090:ll_file_open()) START file . :VOLATILE:0000:734BD555:fd=03:[0x200000405:0x45:0x0](ffff97688b7bb248), flags 574702 00000002:00100000:2.0:1787826427.112735:0:55759:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889cfbc00 x1874670564016640/t8589935289(8589935289) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826443 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1787826427.112773:0:55759:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab18cc from it ffff9768876c9900 00010000:00010000:2.0:1787826427.112778:0:55759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18cc lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83b2f expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.112788:0:55759:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18cc lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83b2f expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826427.112816:0:55759:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x45:0x0](ffff97688b7bb248), inode mode 8180 mask 40 00000080:00200080:2.0:1787826427.112896:0:55759:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=7fff9a213074 00000002:00010000:2.0:1787826427.112920:0:55759:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: open flags 01004040000002 00010000:00010000:2.0:1787826427.112978:0:55759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18d3 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 55759 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.112986:0:55759:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x81000000001000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18d3 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 55759 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1787826427.112997:0:55759:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1787826427.113017:0:55759:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889cfa580 x1874670564016896/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/132176 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1787826427.113026:0:55759:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18d3 lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x81000000000000 pid: 55759 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826427.113050:0:55759:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564016896:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826427.113539:0:55759:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35b40 going to sleep for 11 seconds 00000100:00100000:2.0:1787826427.117157:0:55759:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564016896:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826427.117202:0:55759:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18d3 lrc: 4/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83b44 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.117213:0:55759:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18d3 lrc: 4/1,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83b44 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.117225:0:55759:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18d3 lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83b44 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.117234:0:55759:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18d3 lrc: 4/1,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83b44 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.117252:0:55759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18d3 lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83b44 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.117258:0:55759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18d3 lrc: 5/1,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83b44 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.117263:0:55759:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18d3 lrc: 4/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83b44 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826427.117271:0:55759:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x600002b, status=0 req@ffff976889cfa580 x1874670564016896/t8589935290(8589935290) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826443 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787826427.117281:0:55759:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889cfa580 x1874670564016896/t8589935290(8589935290) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 968/848 e 0 to 0 dl 1787826443 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826427.117295:0:55759:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18d3 lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x22153cd396b83b44 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.117308:0:55759:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 2 res 8589935621/29 (0 0) 00000080:00200000:2.0:1787826427.117313:0:55759:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000002:00100000:2.0:1787826427.117333:0:55759:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976889cfa580 x1874670564016896/t8589935290(8589935290) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826443 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1787826427.117340:0:55759:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab18d3 00000080:00010000:2.0:1787826427.117345:0:55759:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab18d3 from it ffffb2d4422c3cc0 00010000:00010000:2.0:1787826427.117348:0:55759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18d3 lrc: 3/0,1 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83b44 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.117370:0:55759:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18d3 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x22153cd396b83b44 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200080:2.0:1787826427.117449:0:55759:0:(file.c:4655:ll_file_ioctl()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8) cmd=c00c66f3 arg=c28300 00010000:00010000:2.0:1787826427.117501:0:55759:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18d3 lrc: 2/0,0 mode: PW/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0x22153cd396b83b44 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.117514:0:55759:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18d3 lrc: 0/0,0 mode: --/PW res: [0x200000405:0x1d:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x85809400000000 nid: local remote: 0x22153cd396b83b44 expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826427.117526:0:55759:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889cfa580 x1874670564016896/t8589935290(8589935290) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826443 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787826427.117534:0:55759:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787826427.117548:0:55759:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889cfb840 x1874670564017152/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 504/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787826427.117568:0:55759:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564017152:192.168.203.139@tcp:35:lfs.0 00000100:00100000:2.0:1787826427.117661:0:55759:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35b40 going to sleep for 11 seconds 00000100:00100000:1.0:1787826427.121096:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826427.121154:0:42211:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:0.0:1787826427.121189:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585296256 00000100:00100000:0.0:1787826427.121207:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585296256 00000100:00100000:0.0:1787826427.121278:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 633 00000100:00100000:0.0:1787826427.121290:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768895ba450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6522:x1874670585296256:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826427.121312:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18c5 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83b21 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826427.121527:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826427.121535:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768895ba450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6522:x1874670585296256:12345-192.168.203.139@tcp:104: Request processed in 247us (730us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826427.121544:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 633 00010000:00010000:2.0:1787826427.126930:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826427.126952:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18c5 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83b21 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826427.126966:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925ca00 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:2.0:1787826427.126973:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18c5 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83b21 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826427.126989:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18c5 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83b21 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826427.126999:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18c5 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b83b21 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826427.127046:0:42220:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x1d:0x0]: losing layout lock 00010000:00010000:2.0:1787826427.127513:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18c5 lrc: 2/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83b21 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826427.127524:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1787826427.127614:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564017280:192.168.203.139@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1787826427.128582:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18c5 lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83b21 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826427.128597:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ca00/0x8e7c99ee62ab18c5 lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83b21 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826427.128625:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826427.128627:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826427.128630:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826427.132987:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:3.0:1787826427.133258:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564017280:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787826427.133569:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585296384 00000100:00100000:1.0:1787826427.133578:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585296384 00000100:00100000:1.0:1787826427.133677:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 634 00000100:00100000:1.0:1787826427.133688:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aee580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6522:x1874670585296384:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826427.133717:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18cc lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83b2f expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826427.133847:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826427.133855:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aee580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6522:x1874670585296384:12345-192.168.203.139@tcp:104: Request processed in 169us (883us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826427.133863:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 634 00010000:00010000:2.0:1787826427.133929:0:49796:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826427.133939:0:49796:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18cc lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83b2f expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826427.133947:0:49796:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514800 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:2.0:1787826427.133954:0:49796:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18cc lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x45:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83b2f expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826427.133967:0:49796:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18cc lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x45:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83b2f expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826427.133975:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18cc lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x45:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b83b2f expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826427.134015:0:49796:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x45:0x0]: losing layout lock 00010000:00010000:2.0:1787826427.134067:0:49796:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18cc lrc: 2/0,0 mode: --/PR res: [0x200000405:0x45:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83b2f expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826427.134074:0:49796:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826427.134101:0:49796:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18cc lrc: 1/0,0 mode: --/PR res: [0x200000405:0x45:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83b2f expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826427.134110:0:49796:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514800/0x8e7c99ee62ab18cc lrc: 0/0,0 mode: --/PR res: [0x200000405:0x45:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83b2f expref: -99 pid: 55759 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826427.134131:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826427.134133:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826427.134135:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826427.135141:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564017408:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826427.139320:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564017408:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826427.142607:0:55759:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564017152:192.168.203.139@tcp:35:lfs.0 00000002:00100000:2.0:1787826427.142655:0:55759:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889cfa580 x1874670564016896/t8589935290(8589935290) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826443 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1787826427.142707:0:55759:0:(file.c:7076:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000002:00010000:2.0:1787826427.142721:0:55759:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000405:0x1d:0x0], intent: layout flags 0 00010000:00010000:2.0:1787826427.142788:0:55759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18da lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55759 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.142800:0:55759:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18da lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55759 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.142812:0:55759:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18da lrc: 3/1,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55759 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826427.142933:0:55759:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564017536:192.168.203.139@tcp:101:lfs.0 00000100:00100000:2.0:1787826427.143082:0:55759:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35b40 going to sleep for 11 seconds 00000100:00100000:2.0:1787826427.145310:0:55759:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564017536:192.168.203.139@tcp:101:lfs.0 00010000:00010000:2.0:1787826427.145349:0:55759:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18da lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83b60 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.145422:0:55759:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18da lrc: 4/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b60 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826427.145442:0:55759:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff976889cfb840 x1874670564017536/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 376/480 e 0 to 0 dl 1787826443 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826427.145453:0:55759:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18da lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b60 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787826427.145467:0:55759:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18da lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b60 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.145480:0:55759:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 16 res 8589935621/29 (0 0) 00000080:00200000:2.0:1787826427.145487:0:55759:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000405:0x1d:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1787826427.145506:0:55759:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab18da 00000080:00010000:2.0:1787826427.145518:0:55759:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab18da from it ffffb2d4422c3bc8 00010000:00010000:2.0:1787826427.145523:0:55759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18da lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83b60 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826427.145533:0:55759:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18da lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b60 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826427.145563:0:55759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18da lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b60 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826427.145575:0:55759:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18da lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b60 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1787826427.145584:0:55759:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18da lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b60 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826427.145627:0:55759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18da lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83b60 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826427.145640:0:55759:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18da lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b60 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826427.145661:0:55759:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787826427.145674:0:55759:0:(lov_io.c:693:lov_io_mirror_init()) [0x200000405:0x1d:0x0]: flr state: 1, move mirror from 1 to 1, have retried: 0, mirror count: 2 00020000:00200000:2.0:1787826427.145677:0:55759:0:(lov_io.c:750:lov_io_mirror_init()) use delayed RPC state for this IO 00020000:00200000:2.0:1787826427.145685:0:55759:0:(lov_io.c:1253:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:2.0:1787826427.145722:0:55759:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:2.0:1787826427.145736:0:55759:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18da lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b60 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826427.145746:0:55759:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18da lrc: 2/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b60 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1787826427.145753:0:55759:0:(file.c:6959:ll_layout_lock_set()) ### file [0x200000405:0x1d:0x0](000000003d4a00f3) being reconfigured ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18da lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b60 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787826427.145769:0:55759:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787826427.145774:0:55759:0:(lov_io.c:693:lov_io_mirror_init()) [0x200000405:0x1d:0x0]: flr state: 1, move mirror from 1 to 1, have retried: 0, mirror count: 2 00020000:00200000:2.0:1787826427.145776:0:55759:0:(lov_io.c:750:lov_io_mirror_init()) use delayed RPC state for this IO 00020000:00200000:2.0:1787826427.145778:0:55759:0:(lov_io.c:1253:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:2.0:1787826427.145787:0:55759:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787826427.145900:0:55759:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 4294967294 -> 41 00010000:00010000:2.0:1787826427.145906:0:55759:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18da lrc: 3/1,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83b60 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826427.145913:0:55759:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18da lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83b60 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:2.0:1787826427.145955:0:55759:0:(file.c:479:ll_file_release()) START file . :VOLATILE:0000:734BD555:fd=03:[0x200000405:0x45:0x0](ffff97688b7bb248), flags 554002 00000002:00100000:2.0:1787826427.146006:0:55759:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976889cfbc00 x1874670564016640/t8589935289(8589935289) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826443 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787826427.146017:0:55759:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787826427.146023:0:55759:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889cfb840 x1874670564017792/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787826427.146047:0:55759:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564017792:192.168.203.139@tcp:35:lfs.0 00000100:00100000:2.0:1787826427.146217:0:55759:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35e40 going to sleep for 11 seconds 00000100:00100000:2.0:1787826427.150350:0:55759:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb840 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564017792:192.168.203.139@tcp:35:lfs.0 00000002:00100000:2.0:1787826427.150504:0:55759:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976889cfbc00 x1874670564016640/t8589935289(8589935289) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 552/608 e 0 to 0 dl 1787826443 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1787826427.150577:0:55759:0:(file.c:479:ll_file_release()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), flags 154002 00000002:00100000:2.0:1787826427.150645:0:55759:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976882aecf00 x1874670564015104/t8589935285(8589935285) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826443 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787826427.150657:0:55759:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787826427.150662:0:55759:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889cfa580 x1874670564018048/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787826427.150685:0:55759:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564018048:192.168.203.139@tcp:35:lfs.0 00000100:00100000:2.0:1787826427.150877:0:55759:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35e40 going to sleep for 11 seconds 00000100:00100000:2.0:1787826427.156511:0:55759:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55759:1874670564018048:192.168.203.139@tcp:35:lfs.0 00000002:00100000:2.0:1787826427.156570:0:55759:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976882aecf00 x1874670564015104/t8589935285(8589935285) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826443 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826427.182961:0:55761:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826427.183005:0:55761:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826427.183031:0:55761:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826427.183049:0:55761:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826427.183059:0:55761:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826427.183069:0:55761:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826427.183073:0:55761:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826427.183082:0:55761:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4422abb58 00010000:00010000:0.0:1787826427.183086:0:55761:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826427.183096:0:55761:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826427.183116:0:55761:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826427.183126:0:55761:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=24 00000080:00200000:0.0:1787826427.183132:0:55761:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826427.183145:0:55761:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), flags=8 00000080:00200000:0.0:1787826427.183150:0:55761:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f60b.sanity-flr, dir=[0x200000405:0x1c:0x0](ffff9768900e80c8), intent=getattr 00000002:00010000:0.0:1787826427.183166:0:55761:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f60b.sanity-flr,[0x0:0x0:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787826427.183241:0:55761:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab18e1 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 55761 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.183249:0:55761:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab18e1 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55761 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.183260:0:55761:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab18e1 lrc: 3/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 55761 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826427.183294:0:55761:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee1c0 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55761:1874670564018176:192.168.203.139@tcp:101:getfattr.0 00000100:00100000:0.0:1787826427.184712:0:55761:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688741e480 going to sleep for 11 seconds 00000100:00100000:0.0:1787826427.196417:0:55761:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee1c0 pname:cluuid:pid:xid:nid:opc:job getfattr:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55761:1874670564018176:192.168.203.139@tcp:101:getfattr.0 00010000:00010000:0.0:1787826427.196453:0:55761:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab18e1 lrc: 4/1,0 mode: --/CR res: [0x200000bd0:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b83 expref: -99 pid: 55761 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.196483:0:55761:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab18e1 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b83 expref: -99 pid: 55761 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.196493:0:55761:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab18e1 lrc: 4/1,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b83 expref: -99 pid: 55761 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.196508:0:55761:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab18e1 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83b83 expref: -99 pid: 55761 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.196527:0:55761:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab18e1 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b83 expref: -99 pid: 55761 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.196542:0:55761:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab18e1 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83b83 expref: -99 pid: 55761 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.196552:0:55761:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab18e1 lrc: 5/2,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83b83 expref: -99 pid: 55761 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.196561:0:55761:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab18e1 lrc: 4/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83b83 expref: -99 pid: 55761 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826427.196574:0:55761:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff976882aee1c0 x1874670564018176/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 584/720 e 0 to 0 dl 1787826443 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'getfattr.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826427.196587:0:55761:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab18e1 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b83 expref: -99 pid: 55761 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826427.196603:0:55761:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab18e1 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83b83 expref: -99 pid: 55761 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.196616:0:55761:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589935621/29 (0 0) 00000080:00200000:0.0:1787826427.196625:0:55761:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826427.196695:0:55761:0:(namei.c:147:ll_iget()) got inode: [0x200000405:0x1d:0x0](ffff9768901a80c8): rc = 0 00000080:00200000:0.0:1787826427.196715:0:55761:0:(file.c:6854:ll_layout_conf()) [0x200000405:0x1d:0x0]: layout version change: 41 -> 41 00000080:00010000:0.0:1787826427.196722:0:55761:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab18e1 00000080:00010000:0.0:1787826427.196737:0:55761:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:0.0:1787826427.196742:0:55761:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab18e1 from it ffffb2d4422abb20 00010000:00010000:0.0:1787826427.196746:0:55761:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab18e1 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83b83 expref: -99 pid: 55761 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.196756:0:55761:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab18e1 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83b83 expref: -99 pid: 55761 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1787826427.197032:0:55761:0:(file.c:6313:ll_getattr_dentry()) START file f60b.sanity-flr:[0x200000405:0x1d:0x0](ffff9768901a80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826427.197042:0:55761:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8),name=f60b.sanity-flr 00000002:00010000:0.0:1787826427.197079:0:55761:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000405:0x1d:0x0]) in obj [0x200000bd0:0x4:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826427.197093:0:55761:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab18e1 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83b83 expref: -99 pid: 55761 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826427.197106:0:55761:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab18e1 lrc: 2/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83b83 expref: -99 pid: 55761 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826427.197114:0:55761:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd0:0x4:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826427.197119:0:55761:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) 00000080:00010000:0.0:1787826427.197122:0:55761:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000405:0x1d:0x0](ffff9768901a80c8) for lock 0x8e7c99ee62ab18e1 00000080:00010000:0.0:1787826427.197126:0:55761:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab18e1 from it ffffb2d4422abcb8 00010000:00010000:0.0:1787826427.197129:0:55761:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab18e1 lrc: 3/1,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83b83 expref: -99 pid: 55761 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826427.197137:0:55761:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab18e1 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83b83 expref: -99 pid: 55761 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826427.197159:0:55761:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 39 restore needed 0 00000001:00010000:0.0:1787826427.197183:0:55761:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000405:0x1d:0x0] 00000008:00010000:0.0:1787826427.197207:0:55761:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826427.197211:0:55761:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff976889e55ab0, osc lock ffff9768849a2fd8, flags 0x400001002 00010000:00010000:0.0:1787826427.197230:0:55761:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826427.197242:0:55761:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826427.197251:0:55761:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826427.197259:0:55761:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826427.197306:0:55761:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000632bd872/0x8e7c99ee62ab18e8 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55761 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826427.197315:0:55761:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000632bd872/0x8e7c99ee62ab18e8 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55761 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826427.197472:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670564018304:192.168.203.139@tcp:101:getfattr.0 00000020:00200000:0.0:1787826427.197604:0:55761:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1787826427.202735:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826427.203337:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585296640 00000100:00100000:1.0:1787826427.203362:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585296640 00000100:00100000:1.0:1787826427.203434:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 635 00000100:00100000:1.0:1787826427.203446:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aeed00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8545:x1874670585296640:12345-192.168.203.139@tcp:106: 00010000:00010000:1.0:1787826427.203466:0:53061:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826427.203484:0:53061:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:1.0:1787826427.203519:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@000000000980f074[0x110, 1, [0x200000405:0x44:0x0] hash] 00000020:00010000:1.0:1787826427.203522:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826426 blocks: 0 00000020:00010000:1.0:1787826427.203526:0:53061:0:(cl_object.c:322:cl_object_glimpse()) header@0000000071b7e44a[0x116, 2, [0x2c0000401:0x10f:0x0]] 00000020:00010000:1.0:1787826427.203528:0:53061:0:(cl_object.c:322:cl_object_glimpse()) size: 0 mtime: 1787826412 atime: 1787826412 ctime: 1787826426 blocks: 0 00000100:00100000:1.0:1787826427.203606:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aeed00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8545:x1874670585296640:12345-192.168.203.139@tcp:106: Request processed in 162us (887us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826427.203614:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 635 00010000:00010000:1.0:1787826427.213079:0:2402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000632bd872/0x8e7c99ee62ab18e8 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 55761 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826427.213109:0:2402:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000632bd872/0x8e7c99ee62ab18e8 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 55761 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826427.213123:0:2402:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000632bd872/0x8e7c99ee62ab18e8 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55761 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826427.213132:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000632bd872/0x8e7c99ee62ab18e8 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55761 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826427.213139:0:2402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000632bd872/0x8e7c99ee62ab18e8 lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55761 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1787826427.213179:0:2402:0:(osc_lock.c:192:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00000020:00200000:1.0:1787826427.213188:0:2402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1787826427.213203:0:2402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000632bd872/0x8e7c99ee62ab18e8 lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55761 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826427.213211:0:2402:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000632bd872/0x8e7c99ee62ab18e8 lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 55761 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826427.213219:0:2402:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000632bd872/0x8e7c99ee62ab18e8 lrc: 4/0,0 mode: --/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 55761 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826427.213226:0:2402:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925d800 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:1.0:1787826427.213233:0:2402:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000632bd872/0x8e7c99ee62ab18e8 lrc: 5/0,0 mode: --/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 55761 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826427.213246:0:2402:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1787826427.213254:0:2402:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000632bd872/0x8e7c99ee62ab18e8 lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55761 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826427.213262:0:2402:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000632bd872/0x8e7c99ee62ab18e8 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 55761 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826427.213286:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670564018304:192.168.203.139@tcp:101:getfattr.0 00000080:00200000:0.0:1787826427.213501:0:55761:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000405:0x1d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787826427.213509:0:55761:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9768ad1fd8b0 (ffff976890d10098/0) at cl_lock_release():245 00000020:00010000:0.0:1787826427.213513:0:55761:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9768ad1fd8b0 (ffff976890d10098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826427.213519:0:55761:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976889e55ab0 (ffff976890d103f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826427.213523:0:55761:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9768ad1fd8b0 (ffff976890d10098/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826427.213526:0:55761:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976889e55ab0 (ffff976890d10098/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826427.213556:0:55761:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 0/1, layout version 41 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826427.213611:0:55761:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826427.213641:0:55761:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826427.213662:0:55761:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826427.213676:0:55761:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826427.213684:0:55761:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826427.213692:0:55761:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826427.213695:0:55761:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826427.213702:0:55761:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4422abbf8 00010000:00010000:0.0:1787826427.213704:0:55761:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826427.213713:0:55761:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826427.213729:0:55761:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826427.213736:0:55761:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:0.0:1787826427.213743:0:55761:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826427.213747:0:55761:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=9 00000080:00200000:0.0:1787826427.213756:0:55761:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:0.0:1787826427.213762:0:55761:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200000:0.0:1787826427.213784:0:55761:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826427.214234:0:55761:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826427.214245:0:55761:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826427.214319:0:55761:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826427.214328:0:55761:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826427.214334:0:55761:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826427.214337:0:55761:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:1.0:1787826427.214341:0:55761:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4422abbf8 00010000:00010000:1.0:1787826427.214355:0:55761:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826427.214363:0:55761:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787826427.214375:0:55761:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826427.214380:0:55761:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=25 00000080:00200000:1.0:1787826427.214385:0:55761:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787826427.214388:0:55761:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=9 00000080:00200000:1.0:1787826427.214400:0:55761:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), inode mode 81a4 mask 4 00000080:00200000:1.0:1787826427.214404:0:55761:0:(xattr.c:760:ll_xattr_get()) VFS Op:inode=[0x200000405:0x1d:0x0](ffff9768901a80c8), xattr lov 00000080:00200000:2.0:1787826427.239894:0:55765:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:2.0:1787826427.239938:0:55765:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826427.239962:0:55765:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826427.239982:0:55765:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826427.239991:0:55765:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826427.240002:0:55765:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:2.0:1787826427.240006:0:55765:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:2.0:1787826427.240015:0:55765:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4422c3bf8 00010000:00010000:2.0:1787826427.240019:0:55765:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826427.240029:0:55765:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826427.240047:0:55765:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:2.0:1787826427.240057:0:55765:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=27 00000080:00200000:2.0:1787826427.240064:0:55765:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787826427.240068:0:55765:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f60b.sanity-flr, flags=267 00000080:00200000:2.0:1787826427.240098:0:55765:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:2.0:1787826427.240106:0:55765:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826427.240113:0:55765:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826427.240121:0:55765:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826427.240128:0:55765:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826427.240132:0:55765:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:2.0:1787826427.240134:0:55765:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:2.0:1787826427.240138:0:55765:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d4422c3bf8 00010000:00010000:2.0:1787826427.240142:0:55765:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787826427.240155:0:55765:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787826427.240164:0:55765:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:2.0:1787826427.240178:0:55765:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60b.sanity-flr, flags=267 00000080:00200000:2.0:1787826427.240183:0:55765:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8), inode mode 41ed mask 44 00000080:00200008:2.0:1787826427.240189:0:55765:0:(file.c:1090:ll_file_open()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00000080:00200080:2.0:1787826427.240958:0:55765:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000405:0x1c:0x0](ffff9768900e80c8) cmd=c0086915 arg=25c73d0 00000100:00100000:2.0:1787826427.241054:0:55765:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55765:1874670564018432:192.168.203.139@tcp:34:lfs.0 00000100:00100000:2.0:1787826427.241188:0:55765:0:(client.c:2715:ptlrpc_set_wait()) set ffff976890e35e40 going to sleep for 11 seconds 00000100:00100000:2.0:1787826427.247942:0:55765:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:55765:1874670564018432:192.168.203.139@tcp:34:lfs.0 00000080:00200008:2.0:1787826427.248208:0:55765:0:(file.c:479:ll_file_release()) START file d60b.sanity-flr:[0x200000405:0x1c:0x0](ffff9768900e80c8), flags 304000 00010000:00010000:2.0:1787826427.248260:0:55765:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9600/0x8e7c99ee62ab0c1c lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b818c9 expref: -99 pid: 55483 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:2.0:1787826427.361929:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826428.387773:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787826429.414640:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826429.668768:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:307626ff-a376-4faf-93e0-9f55b723d964:2404:1874670564018560:192.168.203.139@tcp:400:kworker.0 00000100:00100000:2.0:1787826429.668884:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564018816:192.168.203.139@tcp:400:kworker.0 00000100:00100000:2.0:1787826429.668913:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564019072:192.168.203.139@tcp:400:kworker.0 00000100:00100000:2.0:1787826429.668937:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1787826429.668942:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564018688:192.168.203.139@tcp:400:kworker.0 00000100:00100000:2.0:1787826429.668964:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564018944:192.168.203.139@tcp:400:kworker.0 00000100:00100000:3.0:1787826429.679064:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564019072:192.168.203.139@tcp:400:kworker.0 00000100:00100000:3.0:1787826429.688274:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564018688:192.168.203.139@tcp:400:kworker.0 00000100:00100000:3.0:1787826429.688293:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564018944:192.168.203.139@tcp:400:kworker.0 00000100:00100000:3.0:1787826429.688304:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:307626ff-a376-4faf-93e0-9f55b723d964:2404:1874670564018560:192.168.203.139@tcp:400:kworker.0 00000100:00100000:3.0:1787826429.688314:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564018816:192.168.203.139@tcp:400:kworker.0 00010000:00010000:2.0:1787826430.431080:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826431.456422:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826432.348865:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826432.348909:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585299328 00000100:00100000:0.0:1787826432.348921:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585299328 00000100:00100000:0.0:1787826432.348984:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 636 00000100:00100000:0.0:1787826432.349080:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff976882aed680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30695:x1874670585299328:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826432.349106:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000030000 nid: local remote: 0x22153cd396b83a87 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826432.349501:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826432.349510:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff976882aed680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:30695:x1874670585299328:12345-192.168.203.139@tcp:104: Request processed in 437us (692us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826432.349520:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 636 00010000:00010000:2.0:1787826432.352383:0:49796:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826432.352404:0:49796:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000030000 nid: local remote: 0x22153cd396b83a87 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826432.352423:0:49796:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976883514c00 already unused, calling callback (ffffffffc0fb7410) 00010000:00010000:2.0:1787826432.352431:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400030000 nid: local remote: 0x22153cd396b83a87 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826432.352507:0:49796:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 2/0,0 mode: --/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400030020 nid: local remote: 0x22153cd396b83a87 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826432.352516:0:49796:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826432.352552:0:49796:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 1/0,0 mode: --/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400030020 nid: local remote: 0x22153cd396b83a87 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826432.352562:0:49796:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9768877a4800 lock: 000000009fa90cbd/0x8e7c99ee62ab189b lrc: 0/0,0 mode: --/PW res: [0x280000401:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400030020 nid: local remote: 0x22153cd396b83a87 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826432.352596:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826432.352598:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826432.352602:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826432.352604:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564019200:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826432.373666:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564019200:192.168.203.139@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1787826432.482282:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826433.504037:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1787826433.732401:0:56330:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 06:27:12 (1787826432) 00010000:00010000:1.0:1787826434.530730:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826434.783292:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670564019456:192.168.203.139@tcp:400:kworker.0 00000100:00100000:1.0:1787826434.783391:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0163840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670564019712:192.168.203.139@tcp:400:kworker.0 00000100:00100000:1.0:1787826434.783421:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:1.0:1787826434.783426:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b01621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:307626ff-a376-4faf-93e0-9f55b723d964:2402:1874670564019328:192.168.203.139@tcp:400:kworker.0 00000100:00100000:1.0:1787826434.783448:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0162940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670564019584:192.168.203.139@tcp:400:kworker.0 00000100:00100000:1.0:1787826434.783467:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0163480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670564019840:192.168.203.139@tcp:400:kworker.0 00000100:00100000:1.0:1787826434.787812:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b01621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:307626ff-a376-4faf-93e0-9f55b723d964:2402:1874670564019328:192.168.203.139@tcp:400:kworker.0 00000100:00100000:1.0:1787826434.787876:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0162940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670564019584:192.168.203.139@tcp:400:kworker.0 00000100:00100000:1.0:1787826434.787886:0:2402:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9768877a4800: committing for last_committed 8589935296 gen 1 00000100:00100000:1.0:1787826434.787918:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0160000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670564019456:192.168.203.139@tcp:400:kworker.0 00000100:00100000:0.0:1787826434.788796:0:2402:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff9768877a4800: committing for last_committed 17179869689 gen 1 00000100:00100000:0.0:1787826434.788840:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0163480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670564019840:192.168.203.139@tcp:400:kworker.0 00000100:00100000:0.0:1787826434.788853:0:2402:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9768877a4800: committing for last_committed 21474837030 gen 1 00000100:00100000:0.0:1787826434.788868:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0163840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670564019712:192.168.203.139@tcp:400:kworker.0 00000080:00200000:0.0:1787826435.093659:0:56416:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826435.093685:0:56416:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826435.093704:0:56416:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826435.093720:0:56416:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826435.093728:0:56416:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826435.093735:0:56416:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826435.093738:0:56416:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826435.093745:0:56416:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d441d03e18 00010000:00010000:0.0:1787826435.093748:0:56416:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826435.093757:0:56416:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826435.093772:0:56416:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 101 00000080:00200000:0.0:1787826435.093780:0:56416:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826435.093786:0:56416:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826435.093792:0:56416:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826435.093799:0:56416:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826435.093806:0:56416:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826435.093809:0:56416:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826435.093811:0:56416:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826435.093815:0:56416:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d441d03be0 00010000:00010000:0.0:1787826435.093818:0:56416:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826435.093954:0:56416:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826435.093962:0:56416:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826435.093974:0:56416:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d61a.sanity-flr, dir=[0x200000007:0x1:0x0](ffff976890119988), flags=1536 00000080:00200000:0.0:1787826435.093977:0:56416:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826435.093984:0:56416:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826435.093990:0:56416:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826435.093997:0:56416:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826435.094004:0:56416:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826435.094008:0:56416:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826435.094011:0:56416:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826435.094014:0:56416:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d441d03ce0 00010000:00010000:0.0:1787826435.094016:0:56416:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826435.094023:0:56416:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826435.094030:0:56416:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 3 00000080:00200000:0.0:1787826435.094035:0:56416:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826435.094040:0:56416:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826435.094046:0:56416:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826435.094053:0:56416:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826435.094059:0:56416:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826435.094062:0:56416:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826435.094065:0:56416:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab09de 00000080:00010000:0.0:1787826435.094068:0:56416:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab09de from it ffffb2d441d03dc8 00010000:00010000:0.0:1787826435.094071:0:56416:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787826435.094076:0:56416:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787826435.094082:0:56416:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 3 00000080:00200000:0.0:1787826435.094086:0:56416:0:(namei.c:2239:do_mkdir()) VFS Op:name=d61a.sanity-flr, dir=[0x200000007:0x1:0x0](ffff976890119988) 00800000:00000004:0.0:1787826435.094106:0:56416:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0001-mdc-ffff9768877a4800: age 1301, max_age 0 00000100:00100000:0.0:1787826435.094177:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1787826435.094199:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670564019968:192.168.203.139@tcp:41:mkdir.0 00000020:00100000:1.0:1787826435.094259:0:56416:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787826435.094271:0:56416:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b0160000 x1874670564020224/t0(0) o36->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 560/2144 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826435.094291:0:56416:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160000 pname:cluuid:pid:xid:nid:opc:job mkdir:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56416:1874670564020224:192.168.203.139@tcp:36:mkdir.0 00000100:00100000:1.0:1787826435.094324:0:56416:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3ee40 going to sleep for 11 seconds 00000100:00100000:1.0:1787826435.099549:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:1.0:1787826435.099592:0:42211:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585302144 00000100:00100000:1.0:1787826435.099601:0:42211:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585302144 00000100:00100000:1.0:1787826435.099646:0:42211:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 637 00000100:00100000:1.0:1787826435.099655:0:42211:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0162940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6524:x1874670585302144:12345-192.168.203.139@tcp:104: 00010000:00010000:1.0:1787826435.099672:0:42211:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787826435.099740:0:42211:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787826435.099747:0:42211:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0162940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6524:x1874670585302144:12345-192.168.203.139@tcp:104: Request processed in 93us (207us total) trans 0 rc 0/0 00000100:00100000:1.0:1787826435.099754:0:42211:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 637 00000100:00100000:0.0:1787826435.100694:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670564019968:192.168.203.139@tcp:41:mkdir.0 00010000:00010000:3.0:1787826435.101037:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826435.101046:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826435.101055:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff976890fdf800 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826435.101060:0:42221:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826435.101068:0:42221:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826435.101077:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826435.101140:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826435.101148:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826435.101167:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826435.101175:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976890fdf800/0x8e7c99ee62ab09de lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x22153cd396b812d3 expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787826435.101192:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826435.101193:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826435.101195:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826435.101207:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564020352:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826435.108994:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564020352:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826435.115107:0:56416:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0160000 pname:cluuid:pid:xid:nid:opc:job mkdir:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56416:1874670564020224:192.168.203.139@tcp:36:mkdir.0 00000080:00200000:2.0:1787826435.115166:0:56416:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000404:0x13:0x0](ffff97688b7bdb88) 00000080:00200000:2.0:1787826435.115173:0:56416:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000404:0x13:0x0], updating i_size 4096 00000080:00200000:0.0:1787826435.124077:0:56417:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826435.124107:0:56417:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826435.124184:0:56417:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.124195:0:56417:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.124205:0:56417:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826435.124231:0:56417:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee1c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56417:1874670564020480:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826435.124312:0:56417:0:(client.c:2715:ptlrpc_set_wait()) set ffff976883578240 going to sleep for 11 seconds 00000100:00100000:0.0:1787826435.127048:0:56417:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee1c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56417:1874670564020480:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826435.127074:0:56417:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.127087:0:56417:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.127099:0:56417:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.127110:0:56417:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.127132:0:56417:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.127140:0:56417:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.127148:0:56417:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826435.127163:0:56417:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff976882aee1c0 x1874670564020480/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826451 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826435.127176:0:56417:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.127189:0:56417:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787826435.127195:0:56417:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826435.127219:0:56417:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1787826435.127225:0:56417:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab18ef 00000080:00010000:0.0:1787826435.127230:0:56417:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826435.127232:0:56417:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab18ef from it ffffb2d441d0bb58 00010000:00010000:0.0:1787826435.127236:0:56417:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.127245:0:56417:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826435.127266:0:56417:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826435.127283:0:56417:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d61a.sanity-flr, dir=[0x200000007:0x1:0x0](ffff976890119988), flags=24 00000080:00200000:0.0:1787826435.127288:0:56417:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d61a.sanity-flr, dir=[0x200000007:0x1:0x0](ffff976890119988), intent=0 00000002:00010000:0.0:1787826435.127307:0:56417:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d61a.sanity-flr,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1787826435.127339:0:56417:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18f6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.127346:0:56417:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18f6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.127359:0:56417:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18f6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826435.127377:0:56417:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee1c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56417:1874670564020608:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826435.127476:0:56417:0:(client.c:2715:ptlrpc_set_wait()) set ffff976883578240 going to sleep for 11 seconds 00000100:00100000:0.0:1787826435.141756:0:56417:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee1c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56417:1874670564020608:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826435.141782:0:56417:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18f6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bf3 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.141802:0:56417:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18f6 lrc: 4/1,0 mode: --/PR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bf3 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.141811:0:56417:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18f6 lrc: 4/1,0 mode: --/PR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bf3 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.141895:0:56417:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18f6 lrc: 4/1,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83bf3 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.141907:0:56417:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18f6 lrc: 4/1,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bf3 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.141920:0:56417:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18f6 lrc: 5/2,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83bf3 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.141928:0:56417:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18f6 lrc: 5/2,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83bf3 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.141941:0:56417:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18f6 lrc: 4/1,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83bf3 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826435.141952:0:56417:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff976882aee1c0 x1874670564020608/t0(0) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826451 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826435.141963:0:56417:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18f6 lrc: 3/1,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bf3 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.141973:0:56417:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff976884b01c00 type 13 mode 4 res 9663677444/19 (0 0) 00000002:00010000:0.0:1787826435.141987:0:56417:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000404:0x13:0x0], intent: lookup flags 0 00010000:00010000:0.0:1787826435.142028:0:56417:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab18fd lrc: 3/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.142035:0:56417:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab18fd lrc: 3/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.142043:0:56417:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab18fd lrc: 3/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826435.142061:0:56417:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aecf00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56417:1874670564020736:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826435.142172:0:56417:0:(client.c:2715:ptlrpc_set_wait()) set ffff976883578240 going to sleep for 11 seconds 00000100:00100000:0.0:1787826435.145363:0:56417:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aecf00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56417:1874670564020736:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826435.145384:0:56417:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab18fd lrc: 4/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83c01 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.145395:0:56417:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab18fd lrc: 4/1,0 mode: --/PR res: [0x240000404:0x13:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83c01 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.145406:0:56417:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab18fd lrc: 4/1,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83c01 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.145415:0:56417:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab18fd lrc: 4/1,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83c01 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.145427:0:56417:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab18fd lrc: 5/2,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83c01 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.145435:0:56417:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab18fd lrc: 5/2,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83c01 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.145443:0:56417:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab18fd lrc: 4/1,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83c01 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826435.145541:0:56417:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff976882aecf00 x1874670564020736/t0(0) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 576/584 e 0 to 0 dl 1787826451 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826435.145555:0:56417:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab18fd lrc: 3/1,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83c01 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.145567:0:56417:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9768ad2e1800 type 13 mode 4 res 9663677444/19 (0 0) 00000080:00200000:0.0:1787826435.145583:0:56417:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826435.145602:0:56417:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000404:0x13:0x0], updating i_size 4096 00000080:00200000:0.0:1787826435.145606:0:56417:0:(namei.c:147:ll_iget()) got inode: [0x240000404:0x13:0x0](ffff97688b7bdb88): rc = 0 00000080:00010000:0.0:1787826435.145610:0:56417:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x240000404:0x13:0x0](ffff97688b7bdb88) for remote lock 0x8e7c99ee62ab18fd 00000080:00010000:0.0:1787826435.145615:0:56417:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000404:0x13:0x0](ffff97688b7bdb88) for lock 0x8e7c99ee62ab18f6 00000080:00010000:0.0:1787826435.145625:0:56417:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000404:0x13:0x0](ffff97688b7bdb88) 00000080:00010000:0.0:1787826435.145629:0:56417:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab18f6 from it ffffb2d441d0ba28 00010000:00010000:0.0:1787826435.145633:0:56417:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18f6 lrc: 3/1,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83bf3 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.145642:0:56417:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18f6 lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bf3 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826435.145653:0:56417:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x8e7c99ee62ab18fd from it ffffb2d441d0ba28 00010000:00010000:0.0:1787826435.145657:0:56417:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab18fd lrc: 3/1,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83c01 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.145664:0:56417:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab18fd lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83c01 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826435.145682:0:56417:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240000404:0x13:0x0](ffff97688b7bdb88), inode mode 41ed mask 1 00000080:00200000:0.0:1787826435.145702:0:56417:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f61a.sanity-flr, dir=[0x240000404:0x13:0x0](ffff97688b7bdb88), flags=8 00000080:00200000:0.0:1787826435.145707:0:56417:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f61a.sanity-flr, dir=[0x240000404:0x13:0x0](ffff97688b7bdb88), intent=getattr 00000002:00010000:0.0:1787826435.145725:0:56417:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f61a.sanity-flr,[0x0:0x0:0x0]) in obj [0x240000404:0x13:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787826435.145758:0:56417:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1904 lrc: 3/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.145765:0:56417:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1904 lrc: 3/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.145774:0:56417:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1904 lrc: 3/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826435.145791:0:56417:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aecf00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56417:1874670564020864:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826435.145913:0:56417:0:(client.c:2715:ptlrpc_set_wait()) set ffff976883578240 going to sleep for 11 seconds 00000100:00100000:0.0:1787826435.150552:0:56417:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aecf00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56417:1874670564020864:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826435.150573:0:56417:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1904 lrc: 4/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.150586:0:56417:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1904 lrc: 4/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.150593:0:56417:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1904 lrc: 4/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.150601:0:56417:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1904 lrc: 3/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.150615:0:56417:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1904 lrc: 4/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.150622:0:56417:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925d600 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:0.0:1787826435.150627:0:56417:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1904 lrc: 4/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.150634:0:56417:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1904 lrc: 4/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.150641:0:56417:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1904 lrc: 5/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.150648:0:56417:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787826435.150655:0:56417:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1904 lrc: 3/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.150812:0:56417:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1904 lrc: 0/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826435.150871:0:56417:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff976882aecf00 x1874670564020864/t0(0) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826451 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826435.150885:0:56417:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240000404:0x13:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787826435.150908:0:56417:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9768ad2e1800 type 13 mode 30 res 9663677444/19 (0 0) 00000080:00200000:0.0:1787826435.151247:0:56417:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826435.151460:0:56417:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826435.151472:0:56417:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826435.151484:0:56417:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826435.151490:0:56417:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826435.151496:0:56417:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826435.151499:0:56417:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab18ef 00000080:00010000:0.0:1787826435.151505:0:56417:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab18ef from it ffffb2d441d0bbf8 00010000:00010000:0.0:1787826435.151508:0:56417:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826435.151517:0:56417:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826435.151530:0:56417:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826435.151537:0:56417:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d61a.sanity-flr, flags=25 00000080:00200000:0.0:1787826435.151543:0:56417:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240000404:0x13:0x0](ffff97688b7bdb88), inode mode 41ed mask 1 00000080:00200000:0.0:1787826435.151555:0:56417:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240000404:0x13:0x0](ffff97688b7bdb88), inode mode 41ed mask 3 00000080:00200000:0.0:1787826435.151559:0:56417:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f61a.sanity-flr, dir=[0x240000404:0x13:0x0](ffff97688b7bdb88), file ffff976883821680, open_flags a141, mode 81b6 opened 0 00000080:00200000:0.0:1787826435.151566:0:56417:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f61a.sanity-flr, dir=[0x240000404:0x13:0x0](ffff97688b7bdb88), intent=open|creat 00000002:00010000:0.0:1787826435.151584:0:56417:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f61a.sanity-flr,[0x240000404:0x14:0x0]) in obj [0x240000404:0x13:0x0], intent: open|creat flags 0120502 00010000:00010000:0.0:1787826435.151623:0:56417:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab190b lrc: 3/0,1 mode: --/CW res: [0x240000404:0x13:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.151630:0:56417:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab190b lrc: 3/0,1 mode: --/CW res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826435.151639:0:56417:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826435.151646:0:56417:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aecf00 x1874670564021120/t0(0) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 528/67888 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826435.151654:0:56417:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab190b lrc: 3/0,1 mode: --/CW res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826435.151671:0:56417:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aecf00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56417:1874670564021120:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826435.151781:0:56417:0:(client.c:2715:ptlrpc_set_wait()) set ffff976883578240 going to sleep for 11 seconds 00000100:00100000:0.0:1787826435.163420:0:56417:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aecf00 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56417:1874670564021120:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826435.163446:0:56417:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab190b lrc: 4/0,1 mode: --/CW res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83c32 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.163470:0:56417:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab190b lrc: 4/0,1 mode: --/PR res: [0x240000404:0x14:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83c32 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.163478:0:56417:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab190b lrc: 4/0,1 mode: --/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83c32 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.163489:0:56417:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab190b lrc: 4/0,1 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83c32 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.163498:0:56417:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab190b lrc: 4/0,1 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83c32 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.163511:0:56417:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab190b lrc: 5/1,1 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83c32 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.163519:0:56417:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab190b lrc: 5/1,1 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83c32 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.163526:0:56417:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab190b lrc: 4/1,0 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83c32 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826435.163538:0:56417:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff976882aecf00 x1874670564021120/t4294967720(4294967720) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 528/608 e 0 to 0 dl 1787826451 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826435.163551:0:56417:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aecf00 x1874670564021120/t4294967720(4294967720) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 528/608 e 0 to 0 dl 1787826451 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826435.163562:0:56417:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab190b lrc: 3/1,0 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83c32 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.163572:0:56417:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9768ad2e1800 type 13 mode 4 res 9663677444/20 (0 0) 00000080:00200000:0.0:1787826435.163579:0:56417:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240000404:0x13:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826435.163591:0:56417:0:(namei.c:1628:ll_create_it()) VFS Op:name=f61a.sanity-flr, dir=[0x240000404:0x13:0x0](ffff97688b7bdb88), intent=open|creat 00000080:00200000:0.0:1787826435.163616:0:56417:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000404:0x14:0x0](ffff9768901a8908) 00000080:00200000:0.0:1787826435.163623:0:56417:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000404:0x14:0x0], updating i_size 0 00000080:00200000:0.0:1787826435.163656:0:56417:0:(file.c:6854:ll_layout_conf()) [0x240000404:0x14:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:0.0:1787826435.163661:0:56417:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240000404:0x14:0x0](ffff9768901a8908) 00000080:00010000:0.0:1787826435.163663:0:56417:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000404:0x14:0x0](ffff9768901a8908) for lock 0x8e7c99ee62ab190b 00000080:00010000:0.0:1787826435.163671:0:56417:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000404:0x14:0x0](ffff9768901a8908) 00000080:00200008:0.0:1787826435.163683:0:56417:0:(file.c:1090:ll_file_open()) START file f61a.sanity-flr:[0x240000404:0x14:0x0](ffff9768901a8908), flags 120501 00000002:00100000:0.0:1787826435.163703:0:56417:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aecf00 x1874670564021120/t4294967720(4294967720) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 528/608 e 0 to 0 dl 1787826451 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787826435.163733:0:56417:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab190b from it ffff976886816240 00010000:00010000:0.0:1787826435.163738:0:56417:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab190b lrc: 3/1,0 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83c32 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.163746:0:56417:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab190b lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83c32 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826435.163764:0:56417:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240000404:0x14:0x0](ffff9768901a8908), inode mode 81a4 mask 40 00000080:00200000:0.0:1787826435.163813:0:56417:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240000404:0x14:0x0](ffff9768901a8908), inode mode 81a4 mask 2 00000080:00200000:0.0:1787826435.163858:0:56417:0:(xattr.c:414:ll_xattr_set()) VFS Op:inode=[0x240000404:0x14:0x0](ffff9768901a8908), xattr lov 00000002:00010000:0.0:1787826435.163874:0:56417:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000404:0x14:0x0]) in obj [0x240000404:0x13:0x0], intent: open flags 040000002 00010000:00010000:0.0:1787826435.164367:0:56417:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1912 lrc: 3/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.164375:0:56417:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1912 lrc: 3/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826435.164385:0:56417:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826435.164391:0:56417:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aee1c0 x1874670564021376/t0(0) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 2216/133424 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826435.164406:0:56417:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1912 lrc: 3/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826435.164423:0:56417:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee1c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56417:1874670564021376:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826435.165551:0:56417:0:(client.c:2715:ptlrpc_set_wait()) set ffff976883578e40 going to sleep for 11 seconds 00000100:00100000:1.0:1787826435.168043:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826435.168123:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585304320 00000100:00100000:0.0:1787826435.168136:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585304320 00000100:00100000:0.0:1787826435.168195:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 638 00000100:00100000:0.0:1787826435.168205:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b01621c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:8247:x1874670585304320:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826435.168222:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab190b lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83c32 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826435.168306:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826435.168313:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b01621c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:8247:x1874670585304320:12345-192.168.203.139@tcp:104: Request processed in 109us (285us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826435.168321:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 638 00010000:00010000:2.0:1787826435.168353:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826435.168362:0:42220:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab190b lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83c32 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826435.168372:0:42220:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925d000 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:2.0:1787826435.168378:0:42220:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab190b lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83c32 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826435.168387:0:42220:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab190b lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83c32 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826435.168396:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab190b lrc: 3/0,0 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b83c32 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826435.168421:0:42220:0:(vvp_object.c:117:vvp_conf_set()) [0x240000404:0x14:0x0]: losing layout lock 00010000:00010000:2.0:1787826435.168508:0:42220:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab190b lrc: 2/0,0 mode: --/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83c32 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826435.168516:0:42220:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826435.168539:0:42220:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab190b lrc: 1/0,0 mode: --/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83c32 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826435.168547:0:42220:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab190b lrc: 0/0,0 mode: --/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83c32 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826435.168584:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826435.168586:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826435.168589:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826435.168615:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564021504:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826435.172059:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564021504:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826435.175545:0:56417:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee1c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56417:1874670564021376:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826435.175566:0:56417:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1912 lrc: 4/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.175578:0:56417:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1912 lrc: 4/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.175586:0:56417:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1912 lrc: 4/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.175593:0:56417:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1912 lrc: 3/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.175602:0:56417:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1912 lrc: 4/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.175607:0:56417:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925d600 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:0.0:1787826435.175612:0:56417:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1912 lrc: 4/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.175618:0:56417:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1912 lrc: 4/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.175625:0:56417:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1912 lrc: 5/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.175631:0:56417:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787826435.175637:0:56417:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1912 lrc: 3/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.175643:0:56417:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1912 lrc: 0/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 56417 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826435.175659:0:56417:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x800002b, status=0 req@ffff976882aee1c0 x1874670564021376/t4294967721(4294967721) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 2216/848 e 0 to 0 dl 1787826451 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826435.175672:0:56417:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aee1c0 x1874670564021376/t4294967721(4294967721) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 2216/848 e 0 to 0 dl 1787826451 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826435.175687:0:56417:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x240000404:0x13:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826435.175728:0:56417:0:(vvp_io.c:1889:vvp_io_init()) [0x240000404:0x14:0x0] ignore/verify layout 1/0, layout version 1 restore needed 0 00000080:00200000:0.0:1787826435.175736:0:56417:0:(vvp_io.c:304:vvp_io_fini()) [0x240000404:0x14:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826435.175771:0:56417:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000404:0x14:0x0], updating i_size 0 00000002:00100000:0.0:1787826435.175788:0:56417:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aee1c0 x1874670564021376/t4294967721(4294967721) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826451 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826435.175820:0:56417:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976882aee1c0 x1874670564021376/t4294967721(4294967721) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826451 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826435.175880:0:56417:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826435.175886:0:56417:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aeda40 x1874670564021760/t0(0) o35->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/2192 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826435.175907:0:56417:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeda40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56417:1874670564021760:192.168.203.139@tcp:35:lfs.0 00000100:00100000:0.0:1787826435.175972:0:56417:0:(client.c:2715:ptlrpc_set_wait()) set ffff976883578e40 going to sleep for 11 seconds 00000100:00100000:0.0:1787826435.186261:0:56417:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeda40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56417:1874670564021760:192.168.203.139@tcp:35:lfs.0 00000002:00100000:0.0:1787826435.186298:0:56417:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976882aee1c0 x1874670564021376/t4294967721(4294967721) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826451 ref 2 fl Complete:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1787826435.186345:0:56417:0:(file.c:479:ll_file_release()) START file f61a.sanity-flr:[0x240000404:0x14:0x0](ffff9768901a8908), flags 100001 00000002:00100000:0.0:1787826435.186394:0:56417:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976882aecf00 x1874670564021120/t4294967720(4294967720) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 528/608 e 0 to 0 dl 1787826451 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826435.186404:0:56417:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826435.186409:0:56417:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aef480 x1874670564022016/t0(0) o35->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/2192 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826435.186427:0:56417:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef480 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56417:1874670564022016:192.168.203.139@tcp:35:lfs.0 00000100:00100000:0.0:1787826435.186490:0:56417:0:(client.c:2715:ptlrpc_set_wait()) set ffff976883578240 going to sleep for 11 seconds 00000100:00100000:0.0:1787826435.188483:0:56417:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef480 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56417:1874670564022016:192.168.203.139@tcp:35:lfs.0 00000002:00100000:0.0:1787826435.188510:0:56417:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976882aecf00 x1874670564021120/t4294967720(4294967720) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 528/608 e 0 to 0 dl 1787826451 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826435.202952:0:56418:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826435.202979:0:56418:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826435.202996:0:56418:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826435.203011:0:56418:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826435.203019:0:56418:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826435.203025:0:56418:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826435.203028:0:56418:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab18ef 00000080:00010000:0.0:1787826435.203035:0:56418:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab18ef from it ffffb2d441d23b58 00010000:00010000:0.0:1787826435.203039:0:56418:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826435.203047:0:56418:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826435.203062:0:56418:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826435.203067:0:56418:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d61a.sanity-flr, flags=24 00000080:00200000:0.0:1787826435.203072:0:56418:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240000404:0x13:0x0](ffff97688b7bdb88), inode mode 41ed mask 1 00000080:00200000:0.0:1787826435.203080:0:56418:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f61a.sanity-flr-2, dir=[0x240000404:0x13:0x0](ffff97688b7bdb88), flags=8 00000080:00200000:0.0:1787826435.203084:0:56418:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f61a.sanity-flr-2, dir=[0x240000404:0x13:0x0](ffff97688b7bdb88), intent=getattr 00000002:00010000:0.0:1787826435.203096:0:56418:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f61a.sanity-flr-2,[0x0:0x0:0x0]) in obj [0x240000404:0x13:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787826435.203154:0:56418:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1919 lrc: 3/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.203165:0:56418:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1919 lrc: 3/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.203174:0:56418:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1919 lrc: 3/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826435.203199:0:56418:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec000 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56418:1874670564022144:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826435.203278:0:56418:0:(client.c:2715:ptlrpc_set_wait()) set ffff976883578240 going to sleep for 11 seconds 00000100:00100000:0.0:1787826435.208097:0:56418:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec000 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56418:1874670564022144:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826435.208127:0:56418:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1919 lrc: 4/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.208141:0:56418:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1919 lrc: 4/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.208148:0:56418:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1919 lrc: 4/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.208155:0:56418:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1919 lrc: 3/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.208165:0:56418:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1919 lrc: 4/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.208171:0:56418:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925d600 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:0.0:1787826435.208176:0:56418:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1919 lrc: 4/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.208183:0:56418:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1919 lrc: 4/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.208191:0:56418:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1919 lrc: 5/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.208197:0:56418:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787826435.208204:0:56418:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1919 lrc: 3/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.208210:0:56418:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1919 lrc: 0/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826435.208227:0:56418:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff976882aec000 x1874670564022144/t0(0) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 592/584 e 0 to 0 dl 1787826451 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826435.208242:0:56418:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240000404:0x13:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787826435.208266:0:56418:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9768ad2e1800 type 13 mode 30 res 9663677444/19 (0 0) 00000080:00200000:0.0:1787826435.208335:0:56418:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:0.0:1787826435.208349:0:56418:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826435.208359:0:56418:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826435.208371:0:56418:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826435.208378:0:56418:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826435.208383:0:56418:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:0.0:1787826435.208386:0:56418:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab18ef 00000080:00010000:0.0:1787826435.208392:0:56418:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab18ef from it ffffb2d441d23bf8 00010000:00010000:0.0:1787826435.208395:0:56418:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826435.208409:0:56418:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1787826435.208423:0:56418:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:0.0:1787826435.208430:0:56418:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d61a.sanity-flr, flags=25 00000080:00200000:0.0:1787826435.208434:0:56418:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240000404:0x13:0x0](ffff97688b7bdb88), inode mode 41ed mask 1 00000080:00200000:0.0:1787826435.208446:0:56418:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240000404:0x13:0x0](ffff97688b7bdb88), inode mode 41ed mask 3 00000080:00200000:0.0:1787826435.208451:0:56418:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f61a.sanity-flr-2, dir=[0x240000404:0x13:0x0](ffff97688b7bdb88), file ffff9768831a6900, open_flags a141, mode 81b6 opened 0 00000080:00200000:0.0:1787826435.208458:0:56418:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f61a.sanity-flr-2, dir=[0x240000404:0x13:0x0](ffff97688b7bdb88), intent=open|creat 00000002:00010000:0.0:1787826435.208595:0:56418:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f61a.sanity-flr-2,[0x240000404:0x15:0x0]) in obj [0x240000404:0x13:0x0], intent: open|creat flags 0120502 00010000:00010000:0.0:1787826435.208635:0:56418:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab1920 lrc: 3/0,1 mode: --/CW res: [0x240000404:0x13:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.208643:0:56418:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab1920 lrc: 3/0,1 mode: --/CW res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826435.208652:0:56418:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826435.208659:0:56418:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aec000 x1874670564022400/t0(0) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 536/67888 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826435.208668:0:56418:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab1920 lrc: 3/0,1 mode: --/CW res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826435.208685:0:56418:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec000 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56418:1874670564022400:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826435.208809:0:56418:0:(client.c:2715:ptlrpc_set_wait()) set ffff976883578240 going to sleep for 11 seconds 00000100:00100000:0.0:1787826435.212712:0:56418:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec000 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56418:1874670564022400:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826435.212737:0:56418:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab1920 lrc: 4/0,1 mode: --/CW res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83c7f expref: -99 pid: 56418 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.212758:0:56418:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab1920 lrc: 4/0,1 mode: --/PR res: [0x240000404:0x15:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83c7f expref: -99 pid: 56418 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.212766:0:56418:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab1920 lrc: 4/0,1 mode: --/PR res: [0x240000404:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83c7f expref: -99 pid: 56418 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.212777:0:56418:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab1920 lrc: 4/0,1 mode: PR/PR res: [0x240000404:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83c7f expref: -99 pid: 56418 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.212787:0:56418:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab1920 lrc: 4/0,1 mode: PR/PR res: [0x240000404:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83c7f expref: -99 pid: 56418 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.212800:0:56418:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab1920 lrc: 5/1,1 mode: PR/PR res: [0x240000404:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83c7f expref: -99 pid: 56418 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.212808:0:56418:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab1920 lrc: 5/1,1 mode: PR/PR res: [0x240000404:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83c7f expref: -99 pid: 56418 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.212815:0:56418:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab1920 lrc: 4/1,0 mode: PR/PR res: [0x240000404:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83c7f expref: -99 pid: 56418 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826435.212866:0:56418:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff976882aec000 x1874670564022400/t4294967724(4294967724) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 536/608 e 0 to 0 dl 1787826451 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826435.212879:0:56418:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aec000 x1874670564022400/t4294967724(4294967724) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 536/608 e 0 to 0 dl 1787826451 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826435.212890:0:56418:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab1920 lrc: 3/1,0 mode: PR/PR res: [0x240000404:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83c7f expref: -99 pid: 56418 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.212901:0:56418:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9768ad2e1800 type 13 mode 4 res 9663677444/21 (0 0) 00000080:00200000:0.0:1787826435.212907:0:56418:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240000404:0x13:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826435.212918:0:56418:0:(namei.c:1628:ll_create_it()) VFS Op:name=f61a.sanity-flr-2, dir=[0x240000404:0x13:0x0](ffff97688b7bdb88), intent=open|creat 00000080:00200000:0.0:1787826435.212944:0:56418:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000404:0x15:0x0](ffff9768901aec08) 00000080:00200000:0.0:1787826435.212955:0:56418:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000404:0x15:0x0], updating i_size 0 00000080:00200000:0.0:1787826435.212988:0:56418:0:(file.c:6854:ll_layout_conf()) [0x240000404:0x15:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:0.0:1787826435.212992:0:56418:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240000404:0x15:0x0](ffff9768901aec08) 00000080:00010000:0.0:1787826435.212996:0:56418:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000404:0x15:0x0](ffff9768901aec08) for lock 0x8e7c99ee62ab1920 00000080:00010000:0.0:1787826435.213004:0:56418:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000404:0x15:0x0](ffff9768901aec08) 00000080:00200008:0.0:1787826435.213017:0:56418:0:(file.c:1090:ll_file_open()) START file f61a.sanity-flr-2:[0x240000404:0x15:0x0](ffff9768901aec08), flags 120501 00000002:00100000:0.0:1787826435.213049:0:56418:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aec000 x1874670564022400/t4294967724(4294967724) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 536/608 e 0 to 0 dl 1787826451 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787826435.213081:0:56418:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1920 from it ffff976886816f00 00010000:00010000:0.0:1787826435.213085:0:56418:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab1920 lrc: 3/1,0 mode: PR/PR res: [0x240000404:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83c7f expref: -99 pid: 56418 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.213094:0:56418:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab1920 lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83c7f expref: -99 pid: 56418 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826435.213126:0:56418:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240000404:0x15:0x0](ffff9768901aec08), inode mode 81a4 mask 40 00000080:00200000:0.0:1787826435.213176:0:56418:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240000404:0x15:0x0](ffff9768901aec08), inode mode 81a4 mask 2 00000080:00200000:0.0:1787826435.213185:0:56418:0:(xattr.c:414:ll_xattr_set()) VFS Op:inode=[0x240000404:0x15:0x0](ffff9768901aec08), xattr lov 00000002:00010000:0.0:1787826435.213201:0:56418:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000404:0x15:0x0]) in obj [0x240000404:0x13:0x0], intent: open flags 040000002 00010000:00010000:0.0:1787826435.213243:0:56418:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1927 lrc: 3/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.213254:0:56418:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1927 lrc: 3/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826435.213264:0:56418:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826435.213269:0:56418:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aec780 x1874670564022656/t0(0) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 2216/133424 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826435.213278:0:56418:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1927 lrc: 3/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826435.213295:0:56418:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56418:1874670564022656:192.168.203.139@tcp:101:lfs.0 00000100:00100000:0.0:1787826435.213443:0:56418:0:(client.c:2715:ptlrpc_set_wait()) set ffff976883578e40 going to sleep for 11 seconds 00000100:00100000:1.0:1787826435.216100:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826435.216207:0:42210:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585304448 00000100:00100000:0.0:1787826435.216216:0:42210:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585304448 00000100:00100000:0.0:1787826435.216335:0:42210:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 639 00000100:00100000:0.0:1787826435.216345:0:42210:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768b0163840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6519:x1874670585304448:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826435.216364:0:42210:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab1920 lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83c7f expref: -99 pid: 56418 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826435.216438:0:42210:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826435.216449:0:42210:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768b0163840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:6519:x1874670585304448:12345-192.168.203.139@tcp:104: Request processed in 101us (354us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826435.216457:0:42210:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 639 00010000:00010000:2.0:1787826435.216478:0:49796:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826435.216487:0:49796:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab1920 lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x22153cd396b83c7f expref: -99 pid: 56418 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826435.216497:0:49796:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925c600 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:2.0:1787826435.216502:0:49796:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab1920 lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x15:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83c7f expref: -99 pid: 56418 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826435.216511:0:49796:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab1920 lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x15:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x22153cd396b83c7f expref: -99 pid: 56418 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826435.216519:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab1920 lrc: 3/0,0 mode: PR/PR res: [0x240000404:0x15:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x22153cd396b83c7f expref: -99 pid: 56418 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826435.216543:0:49796:0:(vvp_object.c:117:vvp_conf_set()) [0x240000404:0x15:0x0]: losing layout lock 00010000:00010000:2.0:1787826435.216579:0:49796:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab1920 lrc: 2/0,0 mode: --/PR res: [0x240000404:0x15:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83c7f expref: -99 pid: 56418 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826435.216585:0:49796:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826435.216613:0:49796:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab1920 lrc: 1/0,0 mode: --/PR res: [0x240000404:0x15:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83c7f expref: -99 pid: 56418 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826435.216621:0:49796:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c600/0x8e7c99ee62ab1920 lrc: 0/0,0 mode: --/PR res: [0x240000404:0x15:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x22153cd396b83c7f expref: -99 pid: 56418 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826435.216637:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826435.216640:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826435.216641:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826435.216649:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1787826435.216662:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564022784:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787826435.220756:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564022784:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826435.220781:0:56418:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec780 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56418:1874670564022656:192.168.203.139@tcp:101:lfs.0 00010000:00010000:0.0:1787826435.220804:0:56418:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1927 lrc: 4/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.220817:0:56418:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1927 lrc: 4/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.220845:0:56418:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1927 lrc: 4/1,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.220851:0:56418:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1927 lrc: 3/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.220857:0:56418:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1927 lrc: 4/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.220861:0:56418:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925d000 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:0.0:1787826435.220864:0:56418:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1927 lrc: 4/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.220869:0:56418:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1927 lrc: 4/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.220875:0:56418:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1927 lrc: 5/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.220879:0:56418:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787826435.220884:0:56418:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1927 lrc: 3/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.220887:0:56418:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d000/0x8e7c99ee62ab1927 lrc: 0/0,0 mode: --/CR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 56418 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826435.220899:0:56418:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x800002b, status=0 req@ffff976882aec780 x1874670564022656/t4294967725(4294967725) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 2216/848 e 0 to 0 dl 1787826451 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826435.220913:0:56418:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aec780 x1874670564022656/t4294967725(4294967725) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 2216/848 e 0 to 0 dl 1787826451 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826435.220924:0:56418:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x240000404:0x13:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826435.220958:0:56418:0:(vvp_io.c:1889:vvp_io_init()) [0x240000404:0x15:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787826435.220963:0:56418:0:(vvp_io.c:304:vvp_io_fini()) [0x240000404:0x15:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826435.220993:0:56418:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000404:0x15:0x0], updating i_size 0 00000002:00100000:0.0:1787826435.221005:0:56418:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff976882aec780 x1874670564022656/t4294967725(4294967725) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826451 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826435.221030:0:56418:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976882aec780 x1874670564022656/t4294967725(4294967725) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826451 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826435.221036:0:56418:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826435.221039:0:56418:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aed2c0 x1874670564023040/t0(0) o35->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/2192 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826435.221054:0:56418:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed2c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56418:1874670564023040:192.168.203.139@tcp:35:lfs.0 00000100:00100000:0.0:1787826435.221101:0:56418:0:(client.c:2715:ptlrpc_set_wait()) set ffff976883578e40 going to sleep for 11 seconds 00000100:00100000:0.0:1787826435.223492:0:56418:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed2c0 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56418:1874670564023040:192.168.203.139@tcp:35:lfs.0 00000002:00100000:0.0:1787826435.223521:0:56418:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976882aec780 x1874670564022656/t4294967725(4294967725) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 760/848 e 0 to 0 dl 1787826451 ref 2 fl Complete:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1787826435.223571:0:56418:0:(file.c:479:ll_file_release()) START file f61a.sanity-flr-2:[0x240000404:0x15:0x0](ffff9768901aec08), flags 100001 00000002:00100000:0.0:1787826435.223613:0:56418:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976882aec000 x1874670564022400/t4294967724(4294967724) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 536/608 e 0 to 0 dl 1787826451 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826435.223623:0:56418:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826435.223629:0:56418:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976882aecb40 x1874670564023296/t0(0) o35->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/2192 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826435.223647:0:56418:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aecb40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56418:1874670564023296:192.168.203.139@tcp:35:lfs.0 00000100:00100000:0.0:1787826435.223742:0:56418:0:(client.c:2715:ptlrpc_set_wait()) set ffff976883578240 going to sleep for 11 seconds 00000100:00100000:0.0:1787826435.225794:0:56418:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aecb40 pname:cluuid:pid:xid:nid:opc:job lfs:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56418:1874670564023296:192.168.203.139@tcp:35:lfs.0 00000002:00100000:0.0:1787826435.225961:0:56418:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff976882aec000 x1874670564022400/t4294967724(4294967724) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 536/608 e 0 to 0 dl 1787826451 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1787826435.227106:0:56198:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826435.227135:0:56198:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826435.227149:0:56198:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787826435.227159:0:56198:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1787826435.227165:0:56198:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826435.227170:0:56198:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826435.227174:0:56198:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab18ef 00000080:00010000:1.0:1787826435.227181:0:56198:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab18ef from it ffffb2d44212bbf8 00010000:00010000:1.0:1787826435.227185:0:56198:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787826435.227194:0:56198:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1787826435.227209:0:56198:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826435.227216:0:56198:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d61a.sanity-flr, flags=25 00000080:00200000:1.0:1787826435.227221:0:56198:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240000404:0x13:0x0](ffff97688b7bdb88), inode mode 41ed mask 1 00000080:00200000:1.0:1787826435.227231:0:56198:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240000404:0x13:0x0](ffff97688b7bdb88), inode mode 41ed mask 3 00000080:00200000:1.0:1787826435.227234:0:56198:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f61a.sanity-flr, dir=[0x240000404:0x13:0x0](ffff97688b7bdb88), file ffff97688a0c4a80, open_flags 8241, mode 81b6 opened 0 00000080:00200000:1.0:1787826435.227240:0:56198:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f61a.sanity-flr, dir=[0x240000404:0x13:0x0](ffff97688b7bdb88), intent=open|creat 00000002:00010000:1.0:1787826435.227256:0:56198:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f61a.sanity-flr,[0x240000404:0x16:0x0]) in obj [0x240000404:0x13:0x0], intent: open|creat flags 0101102 00010000:00010000:1.0:1787826435.227301:0:56198:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab192e lrc: 3/0,1 mode: --/CW res: [0x240000404:0x13:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 56198 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826435.227314:0:56198:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab192e lrc: 3/0,1 mode: --/CW res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 56198 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1787826435.227324:0:56198:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1787826435.227331:0:56198:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9768b0163480 x1874670564023552/t0(0) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 2224/67888 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1787826435.227341:0:56198:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab192e lrc: 3/0,1 mode: --/CW res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 56198 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826435.227359:0:56198:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0163480 pname:cluuid:pid:xid:nid:opc:job bash:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56198:1874670564023552:192.168.203.139@tcp:101:bash.0 00000100:00100000:1.0:1787826435.227560:0:56198:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3ea80 going to sleep for 11 seconds 00000100:00100000:0.0:1787826435.230475:0:56198:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0163480 pname:cluuid:pid:xid:nid:opc:job bash:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56198:1874670564023552:192.168.203.139@tcp:101:bash.0 00010000:00010000:0.0:1787826435.230499:0:56198:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab192e lrc: 4/0,1 mode: --/CW res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83cb7 expref: -99 pid: 56198 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.230519:0:56198:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab192e lrc: 4/0,1 mode: --/PR res: [0x240000404:0x14:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83cb7 expref: -99 pid: 56198 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.230528:0:56198:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab192e lrc: 4/0,1 mode: --/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83cb7 expref: -99 pid: 56198 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.230539:0:56198:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab192e lrc: 4/0,1 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83cb7 expref: -99 pid: 56198 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.230551:0:56198:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab192e lrc: 4/0,1 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83cb7 expref: -99 pid: 56198 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.230563:0:56198:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab192e lrc: 5/1,1 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83cb7 expref: -99 pid: 56198 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.230571:0:56198:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab192e lrc: 5/1,1 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83cb7 expref: -99 pid: 56198 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.230580:0:56198:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab192e lrc: 4/1,0 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83cb7 expref: -99 pid: 56198 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826435.230591:0:56198:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff9768b0163480 x1874670564023552/t4294967728(4294967728) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 2224/848 e 0 to 0 dl 1787826451 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826435.230604:0:56198:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9768b0163480 x1874670564023552/t4294967728(4294967728) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 2224/848 e 0 to 0 dl 1787826451 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826435.230616:0:56198:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 240 ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab192e lrc: 3/1,0 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83cb7 expref: -99 pid: 56198 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826435.230629:0:56198:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab192e lrc: 3/1,0 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83cb7 expref: -99 pid: 56198 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.230639:0:56198:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9768ad2e1800 type 13 mode 4 res 9663677444/20 (0 0) 00000080:00200000:0.0:1787826435.230644:0:56198:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240000404:0x13:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826435.230686:0:56198:0:(namei.c:147:ll_iget()) got inode: [0x240000404:0x14:0x0](ffff9768901a8908): rc = 0 00000080:00200000:0.0:1787826435.230698:0:56198:0:(file.c:6854:ll_layout_conf()) [0x240000404:0x14:0x0]: layout version change: 4294967294 -> 2 00000080:00010000:0.0:1787826435.230703:0:56198:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000404:0x14:0x0](ffff9768901a8908) for lock 0x8e7c99ee62ab192e 00000080:00010000:0.0:1787826435.230714:0:56198:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000404:0x14:0x0](ffff9768901a8908) 00000080:00200008:0.0:1787826435.230735:0:56198:0:(file.c:1090:ll_file_open()) START file f61a.sanity-flr:[0x240000404:0x14:0x0](ffff9768901a8908), flags 101101 00000002:00100000:0.0:1787826435.230761:0:56198:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9768b0163480 x1874670564023552/t4294967728(4294967728) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 768/848 e 0 to 0 dl 1787826451 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787826435.230798:0:56198:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab192e from it ffff976885d9c540 00010000:00010000:0.0:1787826435.230803:0:56198:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab192e lrc: 3/1,0 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83cb7 expref: -99 pid: 56198 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.230818:0:56198:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab192e lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83cb7 expref: -99 pid: 56198 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826435.230905:0:56198:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240000404:0x14:0x0](ffff9768901a8908), inode mode 81a4 mask 42 00000002:00010000:0.0:1787826435.230930:0:56198:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000404:0x14:0x0], intent: getxattr flags 0 00010000:00010000:0.0:1787826435.230961:0:56198:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1935 lrc: 3/1,0 mode: --/PR res: [0x240000404:0x14:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 56198 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.231032:0:56198:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1935 lrc: 3/1,0 mode: --/PR res: [0x240000404:0x14:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT flags: 0x0 pid: 56198 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826435.231042:0:56198:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1935 lrc: 3/1,0 mode: --/PR res: [0x240000404:0x14:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT flags: 0x0 pid: 56198 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826435.231062:0:56198:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job bash:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56198:1874670564023680:192.168.203.139@tcp:101:bash.0 00000100:00100000:0.0:1787826435.231210:0:56198:0:(client.c:2715:ptlrpc_set_wait()) set ffff976883578240 going to sleep for 11 seconds 00000100:00100000:2.0:1787826435.233634:0:56198:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job bash:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56198:1874670564023680:192.168.203.139@tcp:101:bash.0 00010000:00010000:2.0:1787826435.233659:0:56198:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1935 lrc: 4/1,0 mode: --/PR res: [0x240000404:0x14:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83ccc expref: -99 pid: 56198 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.233674:0:56198:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1935 lrc: 4/1,0 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83ccc expref: -99 pid: 56198 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.233685:0:56198:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1935 lrc: 4/1,0 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83ccc expref: -99 pid: 56198 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826435.233700:0:56198:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff976882aee940 x1874670564023680/t0(0) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 568/1336 e 0 to 0 dl 1787826451 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826435.233710:0:56198:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1935 lrc: 3/1,0 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83ccc expref: -99 pid: 56198 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.233722:0:56198:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9768ad2e1800 type 13 mode 4 res 9663677444/20 (0 0) 00000080:00200000:2.0:1787826435.233728:0:56198:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x240000404:0x14:0x0] suppgids 0 -1: rc 0 00000080:00010000:2.0:1787826435.233752:0:56198:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000404:0x14:0x0](ffff9768901a8908) for lock 0x8e7c99ee62ab1935 00000080:00010000:2.0:1787826435.233760:0:56198:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab1935 from it ffffb2d44212ba38 00010000:00010000:2.0:1787826435.233764:0:56198:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1935 lrc: 3/1,0 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83ccc expref: -99 pid: 56198 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.233774:0:56198:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1935 lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83ccc expref: -99 pid: 56198 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1787826435.233810:0:56198:0:(llite_lib.c:2427:ll_setattr_raw()) START file f61a.sanity-flr:[0x240000404:0x14:0x0](ffff9768901a8908) current size 0, valid attrs a068, mode 0, uid 2, gid 0, new size 0, atime 1787826435.000000000, mtime 1787826435.000000000, ctime 1787826435.000000000 00010000:00010000:2.0:1787826435.233867:0:56198:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab192e lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b83cb7 expref: -99 pid: 56198 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826435.233886:0:56198:0:(vvp_object.c:117:vvp_conf_set()) [0x240000404:0x14:0x0]: losing layout lock 00010000:00010000:2.0:1787826435.233929:0:56198:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab192e lrc: 1/0,0 mode: --/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b83cb7 expref: -99 pid: 56198 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826435.233936:0:56198:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826435.233939:0:56198:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdfe00/0x8e7c99ee62ab192e lrc: 0/0,0 mode: --/PR res: [0x240000404:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b83cb7 expref: -99 pid: 56198 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1787826435.234338:0:56198:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787826435.234347:0:56198:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889cf8b40 x1874670564023936/t0(0) o36->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826435.234370:0:56198:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8b40 pname:cluuid:pid:xid:nid:opc:job bash:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56198:1874670564023936:192.168.203.139@tcp:36:bash.0 00000100:00100000:2.0:1787826435.234438:0:56198:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 11 seconds 00000100:00100000:2.0:1787826435.236718:0:56198:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8b40 pname:cluuid:pid:xid:nid:opc:job bash:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56198:1874670564023936:192.168.203.139@tcp:36:bash.0 00000080:00200000:2.0:1787826435.236749:0:56198:0:(file.c:1848:ll_io_set_mirror()) f61a.sanity-flr: desiginated mirror: 0 00000080:00200000:2.0:1787826435.236754:0:56198:0:(vvp_io.c:1889:vvp_io_init()) [0x240000404:0x14:0x0] ignore/verify layout 0/0, layout version 39 restore needed 0 00010000:00010000:2.0:1787826435.236770:0:56198:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9768ad2e1800 type 13 mode 31 res 9663677444/20 (0 0) 00000080:00010000:2.0:1787826435.236777:0:56198:0:(file.c:7076:ll_layout_intent()) ### lustre: requeue layout lock for file [0x240000404:0x14:0x0](ffff9768901a8908) 00000002:00010000:2.0:1787826435.236784:0:56198:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000404:0x14:0x0]) in obj [0x240000404:0x14:0x0], intent: layout flags 0 00010000:00010000:2.0:1787826435.236812:0:56198:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab193c lrc: 3/1,0 mode: --/CR res: [0x240000404:0x14:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 56198 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.236818:0:56198:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab193c lrc: 3/1,0 mode: --/CR res: [0x240000404:0x14:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT flags: 0x0 pid: 56198 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.236850:0:56198:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab193c lrc: 3/1,0 mode: --/CR res: [0x240000404:0x14:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT flags: 0x0 pid: 56198 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826435.236868:0:56198:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8780 pname:cluuid:pid:xid:nid:opc:job bash:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56198:1874670564024064:192.168.203.139@tcp:101:bash.0 00000100:00100000:2.0:1787826435.236986:0:56198:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 11 seconds 00000100:00100000:2.0:1787826435.238783:0:56198:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8780 pname:cluuid:pid:xid:nid:opc:job bash:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56198:1874670564024064:192.168.203.139@tcp:101:bash.0 00010000:00010000:2.0:1787826435.238809:0:56198:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab193c lrc: 4/1,0 mode: CR/CR res: [0x240000404:0x14:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83cda expref: -99 pid: 56198 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.238820:0:56198:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab193c lrc: 4/1,0 mode: CR/CR res: [0x240000404:0x14:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83cda expref: -99 pid: 56198 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826435.238857:0:56198:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff976889cf8780 x1874670564024064/t0(0) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 376/584 e 0 to 0 dl 1787826451 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826435.238867:0:56198:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 240 ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab193c lrc: 3/1,0 mode: CR/CR res: [0x240000404:0x14:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83cda expref: -99 pid: 56198 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787826435.238877:0:56198:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab193c lrc: 3/1,0 mode: CR/CR res: [0x240000404:0x14:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83cda expref: -99 pid: 56198 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.238888:0:56198:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9768ad2e1800 type 13 mode 16 res 9663677444/20 (0 0) 00000080:00200000:2.0:1787826435.238895:0:56198:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x240000404:0x14:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1787826435.238909:0:56198:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000404:0x14:0x0](ffff9768901a8908) for lock 0x8e7c99ee62ab193c 00000080:00010000:2.0:1787826435.238918:0:56198:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab193c from it ffffb2d44212b8c8 00010000:00010000:2.0:1787826435.238922:0:56198:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab193c lrc: 3/1,0 mode: CR/CR res: [0x240000404:0x14:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83cda expref: -99 pid: 56198 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.238930:0:56198:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab193c lrc: 2/0,0 mode: CR/CR res: [0x240000404:0x14:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83cda expref: -99 pid: 56198 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826435.238967:0:56198:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab193c lrc: 2/1,0 mode: CR/CR res: [0x240000404:0x14:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83cda expref: -99 pid: 56198 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826435.238978:0:56198:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab193c lrc: 2/1,0 mode: CR/CR res: [0x240000404:0x14:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83cda expref: -99 pid: 56198 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1787826435.238986:0:56198:0:(file.c:6959:ll_layout_lock_set()) ### file [0x240000404:0x14:0x0](0000000057d1b507) being reconfigured ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab193c lrc: 3/1,0 mode: CR/CR res: [0x240000404:0x14:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83cda expref: -99 pid: 56198 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826435.239021:0:56198:0:(file.c:6854:ll_layout_conf()) [0x240000404:0x14:0x0]: layout version change: 4294967294 -> 2 00010000:00010000:2.0:1787826435.239027:0:56198:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab193c lrc: 3/1,0 mode: CR/CR res: [0x240000404:0x14:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x22153cd396b83cda expref: -99 pid: 56198 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826435.239035:0:56198:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab193c lrc: 2/0,0 mode: CR/CR res: [0x240000404:0x14:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x22153cd396b83cda expref: -99 pid: 56198 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:2.0:1787826435.239055:0:56198:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:2.0:1787826435.239068:0:56198:0:(lov_io.c:1253:lov_io_iter_init()) component[1] flags 0x0 00000080:00200000:2.0:1787826435.239070:0:56198:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:2.0:1787826435.239092:0:56198:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 4 00000008:00010000:2.0:1787826435.239096:0:56198:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff97688757c5b0, osc lock ffff976883f7a270, flags 0x80000000 00010000:00010000:2.0:1787826435.239143:0:56198:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab1943 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 56198 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.239156:0:56198:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x80000000 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab1943 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->1048575] (req 0->1048575) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 56198 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.239166:0:56198:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab1943 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->1048575] (req 0->1048575) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 56198 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787826435.239184:0:56198:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8780 pname:cluuid:pid:xid:nid:opc:job bash:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56198:1874670564024192:192.168.203.139@tcp:101:bash.0 00000100:00100000:2.0:1787826435.239236:0:56198:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 11 seconds 00000100:00100000:2.0:1787826435.243265:0:56198:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8780 pname:cluuid:pid:xid:nid:opc:job bash:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56198:1874670564024192:192.168.203.139@tcp:101:bash.0 00010000:00010000:2.0:1787826435.243302:0:56198:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab1943 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83ce1 expref: -99 pid: 56198 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.243320:0:56198:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab1943 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x10000080000000 nid: local remote: 0x22153cd396b83ce1 expref: -99 pid: 56198 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.243332:0:56198:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab1943 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x80000000 nid: local remote: 0x22153cd396b83ce1 expref: -99 pid: 56198 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.243347:0:56198:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab1943 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x10000080000000 nid: local remote: 0x22153cd396b83ce1 expref: -99 pid: 56198 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787826435.243360:0:56198:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab1943 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x10020080000000 nid: local remote: 0x22153cd396b83ce1 expref: -99 pid: 56198 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.243376:0:56198:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab1943 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020080000000 nid: local remote: 0x22153cd396b83ce1 expref: -99 pid: 56198 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.243393:0:56198:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab1943 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020080000000 nid: local remote: 0x22153cd396b83ce1 expref: -99 pid: 56198 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826435.243492:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787826435.243509:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564024320:192.168.203.139@tcp:10:bash.0 00000100:00100000:3.0:1787826435.246202:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564024320:192.168.203.139@tcp:10:bash.0 00000080:00200000:2.0:1787826435.246376:0:56198:0:(file.c:1775:ll_merge_attr_nolock()) [0x240000404:0x14:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1787826435.246382:0:56198:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976890fb74e8 (ffff976890d10128/0) at cl_lock_release():245 00000020:00010000:2.0:1787826435.246386:0:56198:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976890fb74e8 (ffff976890d10128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787826435.246390:0:56198:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757c5b0 (ffff976890d10a28/1) at cl_lock_cancel():142 00010000:00010000:2.0:1787826435.246399:0:56198:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab1943 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020080000000 nid: local remote: 0x22153cd396b83ce1 expref: -99 pid: 56198 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.246412:0:56198:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab1943 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020080000000 nid: local remote: 0x22153cd396b83ce1 expref: -99 pid: 56198 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1787826435.246427:0:56198:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976890fb74e8 (ffff976890d10128/0) at cl_lock_fini():72 00000020:00010000:2.0:1787826435.246431:0:56198:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757c5b0 (ffff976890d10128/1) at cl_lock_fini():72 00000080:00200000:2.0:1787826435.246449:0:56198:0:(vvp_io.c:304:vvp_io_fini()) [0x240000404:0x14:0x0] ignore/verify layout 0/0, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1787826435.246461:0:56198:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f61a.sanity-flr:[0x240000404:0x14:0x0](ffff9768901a8908) current size 0, valid attrs 1000a168, mode 0, uid 2, gid 0, new size 0, atime 1787826435.000000000, mtime 1787826435.000000000, ctime 1787826435.000000000, rc 0 00000080:00200008:2.0:1787826435.246538:0:56198:0:(file.c:2652:do_file_write_iter()) START file f61a.sanity-flr:[0x240000404:0x14:0x0], ppos: 0, count: 4 00000080:00200000:2.0:1787826435.246550:0:56198:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 4 bytes 00000080:00200000:2.0:1787826435.246556:0:56198:0:(file.c:2628:ll_do_tiny_write()) result: 0, original count 4 00000080:00200000:2.0:1787826435.246561:0:56198:0:(file.c:2090:ll_file_io_generic()) f61a.sanity-flr: write ppos: 0, bytes: 4 00000080:00200000:2.0:1787826435.246565:0:56198:0:(file.c:1848:ll_io_set_mirror()) f61a.sanity-flr: desiginated mirror: 0 00000020:00200000:2.0:1787826435.246571:0:56198:0:(cl_io.c:218:cl_io_rw_init()) header@00000000a1795af2[0x110, 1, [0x240000404:0x14:0x0] hash] 00000020:00200000:2.0:1787826435.246573:0:56198:0:(cl_io.c:218:cl_io_rw_init()) io range: 2 [0, 4) 0 0 00000080:00200000:2.0:1787826435.246577:0:56198:0:(vvp_io.c:1889:vvp_io_init()) [0x240000404:0x14:0x0] ignore/verify layout 0/0, layout version 2 restore needed 0 00000080:00200000:2.0:1787826435.246589:0:56198:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:2.0:1787826435.246596:0:56198:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 4) 4, 4 00020000:00200000:2.0:1787826435.246599:0:56198:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787826435.246612:0:56198:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:2.0:1787826435.246623:0:56198:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1787826435.246627:0:56198:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff97688757c5b0, osc lock ffff976883f7a270, flags 0x0 00010000:00010000:2.0:1787826435.246641:0:56198:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab1943 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x28:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020080000000 nid: local remote: 0x22153cd396b83ce1 expref: -99 pid: 56198 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826435.246650:0:56198:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab1943 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020080000000 nid: local remote: 0x22153cd396b83ce1 expref: -99 pid: 56198 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826435.246657:0:56198:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab1943 lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020080000000 nid: local remote: 0x22153cd396b83ce1 expref: -99 pid: 56198 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826435.246662:0:56198:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab1943 lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020080000000 nid: local remote: 0x22153cd396b83ce1 expref: -99 pid: 56198 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826435.246667:0:56198:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab1943 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020080000000 nid: local remote: 0x22153cd396b83ce1 expref: -99 pid: 56198 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826435.246674:0:56198:0:(vvp_io.c:1339:vvp_io_write_start()) f61a.sanity-flr: write [0, 4) append 0 append lockpos 0 00000080:00200000:2.0:1787826435.246681:0:56198:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 4 bytes 00000080:00200000:2.0:1787826435.246717:0:56198:0:(rw26.c:875:ll_write_end()) pos 0, len 4, copied 4 00000080:00200000:2.0:1787826435.246723:0:56198:0:(rw26.c:933:ll_write_end()) page@0000000030076e3c[3 00000000a6c1a88d 2 1 0000000067cb196d 0 0 0] 00000080:00200000:2.0:1787826435.246726:0:56198:0:(rw26.c:933:ll_write_end()) vmpage @00000000900b1077 17ffffc0001001 3:0 ffff9768ad0e5cb0 0 lru 00000080:00200000:2.0:1787826435.246731:0:56198:0:(rw26.c:933:ll_write_end()) osc-page@000000000da0be80 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000006584ce1d 0000000021df0a93 > 3< 0 0 > 4< 0 0 8 185958400 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787826435.246733:0:56198:0:(rw26.c:933:ll_write_end()) end page@0000000030076e3c 00000080:00200000:2.0:1787826435.246734:0:56198:0:(rw26.c:933:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1787826435.246737:0:56198:0:(vvp_io.c:1214:vvp_io_write_commit()) commit async pages: 1, from 0, to 4 prio 0 00000008:00010000:2.0:1787826435.246751:0:56198:0:(osc_cache.c:858:osc_extent_find()) ### extent: 000000005e3980cb ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab1943 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020080000000 nid: local remote: 0x22153cd396b83ce1 expref: -99 pid: 56198 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826435.246765:0:56198:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff9768901a8ab8, count 1, dirtied 1 00000080:00200000:2.0:1787826435.246767:0:56198:0:(vvp_io.c:1237:vvp_io_write_commit()) Committed 1 pages 4 bytes, tot: 4 00000080:00200000:2.0:1787826435.246769:0:56198:0:(vvp_io.c:1421:vvp_io_write_start()) f61a.sanity-flr: write bytes 0, result: 4 00000020:00010000:2.0:1787826435.246778:0:56198:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff976890fb74e8 (ffff976890d10128/0) at cl_lock_release():245 00000020:00010000:2.0:1787826435.246780:0:56198:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff976890fb74e8 (ffff976890d10128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787826435.246781:0:56198:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff97688757c5b0 (ffff976890d10a28/1) at cl_lock_cancel():142 00010000:00010000:2.0:1787826435.246784:0:56198:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab1943 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020080000000 nid: local remote: 0x22153cd396b83ce1 expref: -99 pid: 56198 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826435.246789:0:56198:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab1943 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020080000000 nid: local remote: 0x22153cd396b83ce1 expref: -99 pid: 56198 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:2.0:1787826435.246795:0:56198:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff976890fb74e8 (ffff976890d10128/0) at cl_lock_fini():72 00000020:00010000:2.0:1787826435.246797:0:56198:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff97688757c5b0 (ffff976890d10128/1) at cl_lock_fini():72 00000080:00200000:2.0:1787826435.246799:0:56198:0:(vvp_io.c:533:vvp_io_advance()) advancing 4 bytes 00000080:00200000:2.0:1787826435.246802:0:56198:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:2.0:1787826435.246806:0:56198:0:(vvp_io.c:304:vvp_io_fini()) [0x240000404:0x14:0x0] ignore/verify layout 0/0, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787826435.246808:0:56198:0:(file.c:2240:ll_file_io_generic()) f61a.sanity-flr: 2 io complete with rc: 0, result: 4, restart: 0 00000080:00200000:2.0:1787826435.246811:0:56198:0:(file.c:1775:ll_merge_attr_nolock()) [0x240000404:0x14:0x0] updating i_size 4 i_blocks 0 00000080:00200000:2.0:1787826435.246814:0:56198:0:(file.c:2326:ll_file_io_generic()) iot: 2, result: 4 00000080:00200008:2.0:1787826435.246856:0:56198:0:(file.c:479:ll_file_release()) START file f61a.sanity-flr:[0x240000404:0x14:0x0](ffff9768901a8908), flags 100001 00000002:00100000:2.0:1787826435.246911:0:56198:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9768b0163480 x1874670564023552/t4294967728(4294967728) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 768/848 e 0 to 0 dl 1787826451 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787826435.246928:0:56198:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787826435.246934:0:56198:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889cfb840 x1874670564024576/t0(0) o35->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/2192 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787826435.246953:0:56198:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb840 pname:cluuid:pid:xid:nid:opc:job bash:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56198:1874670564024576:192.168.203.139@tcp:35:bash.0 00000100:00100000:2.0:1787826435.247103:0:56198:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 11 seconds 00000100:00100000:2.0:1787826435.249626:0:56198:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb840 pname:cluuid:pid:xid:nid:opc:job bash:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56198:1874670564024576:192.168.203.139@tcp:35:bash.0 00000002:00100000:2.0:1787826435.249657:0:56198:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9768b0163480 x1874670564023552/t4294967728(4294967728) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 768/848 e 0 to 0 dl 1787826451 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1787826435.271501:0:56419:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:1.0:1787826435.271526:0:56419:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826435.271543:0:56419:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1787826435.271559:0:56419:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1787826435.271568:0:56419:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787826435.271575:0:56419:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:1.0:1787826435.271578:0:56419:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab18ef 00000080:00010000:1.0:1787826435.271585:0:56419:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab18ef from it ffffb2d441c63b58 00010000:00010000:1.0:1787826435.271590:0:56419:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1787826435.271599:0:56419:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:1.0:1787826435.271614:0:56419:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:1.0:1787826435.271622:0:56419:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d61a.sanity-flr, flags=8 00000080:00200008:1.0:1787826435.271630:0:56419:0:(file.c:6313:ll_getattr_dentry()) START file d61a.sanity-flr:[0x240000404:0x13:0x0](ffff97688b7bdb88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1787826435.271634:0:56419:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240000404:0x13:0x0](ffff97688b7bdb88),name=d61a.sanity-flr 00000002:00010000:1.0:1787826435.271647:0:56419:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000404:0x13:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787826435.271654:0:56419:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9768ad2e1800 type 13 mode 30 res 9663677444/19 (0 0) 00010000:00010000:1.0:1787826435.271694:0:56419:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab194a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 56419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826435.271702:0:56419:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab194a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 56419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787826435.271712:0:56419:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab194a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 56419 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787826435.271736:0:56419:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160f00 pname:cluuid:pid:xid:nid:opc:job chown:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56419:1874670564024704:192.168.203.139@tcp:101:chown.0 00000100:00100000:1.0:1787826435.271848:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3ec00 going to sleep for 11 seconds 00000100:00100000:2.0:1787826435.277401:0:56419:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0160f00 pname:cluuid:pid:xid:nid:opc:job chown:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56419:1874670564024704:192.168.203.139@tcp:101:chown.0 00010000:00010000:2.0:1787826435.277425:0:56419:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab194a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83cef expref: -99 pid: 56419 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.277461:0:56419:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab194a lrc: 4/1,0 mode: --/PR res: [0x240000404:0x13:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83cef expref: -99 pid: 56419 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.277470:0:56419:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab194a lrc: 4/1,0 mode: --/PR res: [0x240000404:0x13:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83cef expref: -99 pid: 56419 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.277481:0:56419:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab194a lrc: 4/1,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83cef expref: -99 pid: 56419 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.277492:0:56419:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab194a lrc: 4/1,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83cef expref: -99 pid: 56419 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.277504:0:56419:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab194a lrc: 5/2,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83cef expref: -99 pid: 56419 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.277512:0:56419:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab194a lrc: 5/2,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83cef expref: -99 pid: 56419 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.277520:0:56419:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab194a lrc: 4/1,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83cef expref: -99 pid: 56419 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787826435.277531:0:56419:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9768b0160f00 x1874670564024704/t0(0) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 576/584 e 0 to 0 dl 1787826451 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chown.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787826435.277544:0:56419:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab194a lrc: 3/1,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83cef expref: -99 pid: 56419 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.277554:0:56419:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9768ad2e1800 type 13 mode 4 res 9663677444/19 (0 0) 00000080:00200000:2.0:1787826435.277561:0:56419:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787826435.277580:0:56419:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000404:0x13:0x0], updating i_size 4096 00000080:00010000:2.0:1787826435.277585:0:56419:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000404:0x13:0x0](ffff97688b7bdb88) for lock 0x8e7c99ee62ab194a 00000080:00010000:2.0:1787826435.277590:0:56419:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000404:0x13:0x0](ffff97688b7bdb88) 00000080:00010000:2.0:1787826435.277593:0:56419:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab194a from it ffffb2d441c63cb8 00010000:00010000:2.0:1787826435.277596:0:56419:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab194a lrc: 3/1,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83cef expref: -99 pid: 56419 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787826435.277605:0:56419:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab194a lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83cef expref: -99 pid: 56419 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787826435.277698:0:56419:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988),name=/ 00000002:00010000:2.0:1787826435.277715:0:56419:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787826435.277726:0:56419:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1787826435.277737:0:56419:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:2.0:1787826435.277744:0:56419:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787826435.277748:0:56419:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) 00000080:00010000:2.0:1787826435.277751:0:56419:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff976890119988) for lock 0x8e7c99ee62ab18ef 00000080:00010000:2.0:1787826435.277755:0:56419:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8e7c99ee62ab18ef from it ffffb2d441c63bf8 00010000:00010000:2.0:1787826435.277758:0:56419:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1787826435.277765:0:56419:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d200/0x8e7c99ee62ab18ef lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83bd7 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:2.0:1787826435.277773:0:56419:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff976890119988), inode mode 41ed mask 1 00000080:00200000:2.0:1787826435.277780:0:56419:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d61a.sanity-flr, flags=9 00000080:00200008:2.0:1787826435.277794:0:56419:0:(llite_lib.c:2427:ll_setattr_raw()) START file d61a.sanity-flr:[0x240000404:0x13:0x0](ffff97688b7bdb88) current size 4096, valid attrs 42, mode ffff, uid 500, gid -26776, new size 18446629073985798144, atime 1787826435.000000000, mtime 1787826435.000000000, ctime 1787826435.000000000 00010000:00010000:2.0:1787826435.277810:0:56419:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab194a lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x22153cd396b83cef expref: -99 pid: 56419 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826435.277860:0:56419:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab18fd lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b83c01 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826435.277897:0:56419:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab194a lrc: 1/0,0 mode: --/PR res: [0x240000404:0x13:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b83cef expref: -99 pid: 56419 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826435.277904:0:56419:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787826435.277907:0:56419:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976890fdec00/0x8e7c99ee62ab194a lrc: 0/0,0 mode: --/PR res: [0x240000404:0x13:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b83cef expref: -99 pid: 56419 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1787826435.277921:0:56419:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787826435.277928:0:56419:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889cfb840 x1874670564024960/t0(0) o36->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826435.277948:0:56419:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb840 pname:cluuid:pid:xid:nid:opc:job chown:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:56419:1874670564024960:192.168.203.139@tcp:36:chown.0 00000100:00100000:2.0:1787826435.278033:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 11 seconds 00000100:00100000:1.0:1787826435.280985:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.139@tcp (source: 12345-192.168.203.139@tcp) 00000100:00100000:0.0:1787826435.281170:0:53061:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:0.0:1787826435.281187:0:53061:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874670585304576 00000100:00100000:0.0:1787826435.281194:0:53061:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874670585304576 00000100:00100000:0.0:1787826435.281246:0:53061:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.139@tcp, seq: 640 00000100:00100000:0.0:1787826435.281254:0:53061:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9768895bb050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:32404:x1874670585304576:12345-192.168.203.139@tcp:104: 00010000:00010000:0.0:1787826435.281271:0:53061:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab18fd lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x22153cd396b83c01 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826435.281343:0:53061:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787826435.281350:0:53061:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9768895bb050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:09ed01c7-7b8c-4e06-9d82-a6b70e42014a+5:32404:x1874670585304576:12345-192.168.203.139@tcp:104: Request processed in 97us (381us total) trans 0 rc 0/0 00000100:00100000:0.0:1787826435.281360:0:53061:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.139@tcp, seq: 640 00010000:00010000:3.0:1787826435.281709:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826435.281717:0:42221:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab18fd lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x22153cd396b83c01 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826435.281736:0:42221:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff97688925c400 already unused, calling callback (ffffffffc18f3990) 00010000:00010000:3.0:1787826435.281741:0:42221:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab18fd lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x10/0x10 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x22153cd396b83c01 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826435.281749:0:42221:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab18fd lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x10/0x10 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x22153cd396b83c01 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826435.281756:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab18fd lrc: 3/0,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x10/0x10 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x22153cd396b83c01 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826435.281807:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab18fd lrc: 2/0,0 mode: --/PR res: [0x240000404:0x13:0x0].0x0 bits 0x10/0x10 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x22153cd396b83c01 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826435.281814:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826435.281866:0:42221:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab18fd lrc: 1/0,0 mode: --/PR res: [0x240000404:0x13:0x0].0x0 bits 0x10/0x10 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x22153cd396b83c01 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826435.281874:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925c400/0x8e7c99ee62ab18fd lrc: 0/0,0 mode: --/PR res: [0x240000404:0x13:0x0].0x0 bits 0x10/0x10 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x22153cd396b83c01 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826435.281894:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826435.281897:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826435.281900:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826435.281909:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826435.281923:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564025088:192.168.203.139@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826435.289053:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2403:1874670564025088:192.168.203.139@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1787826435.551028:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:1.0:1787826435.807037:0:42426:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x19:0x0] ignore/verify layout 1/0, layout version 37 restore needed 0 00020000:00200000:1.0:1787826435.807052:0:42426:0:(lov_io.c:693:lov_io_mirror_init()) [0x200000405:0x19:0x0]: flr state: 1, move mirror from 1 to 1, have retried: 0, mirror count: 2 00020000:00200000:1.0:1787826435.807055:0:42426:0:(lov_io.c:750:lov_io_mirror_init()) use delayed RPC state for this IO 00020000:00200000:1.0:1787826435.807061:0:42426:0:(lov_io.c:1253:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:1.0:1787826435.807621:0:42426:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x19:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1787826435.807633:0:42426:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1787826435.807638:0:42426:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1787826435.807647:0:42426:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:2.0:1787826436.575168:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826437.598976:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826438.624661:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826439.648813:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826439.903019:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670564025344:192.168.203.139@tcp:400:kworker.0 00000100:00100000:0.0:1787826439.903020:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0162d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:307626ff-a376-4faf-93e0-9f55b723d964:2401:1874670564025216:192.168.203.139@tcp:400:kworker.0 00000100:00100000:0.0:1787826439.903126:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b01603c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670564025472:192.168.203.139@tcp:400:kworker.0 00000100:00100000:1.0:1787826439.903128:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0162580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670564025600:192.168.203.139@tcp:400:kworker.0 00000100:00100000:0.0:1787826439.903153:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0161680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670564025728:192.168.203.139@tcp:400:kworker.0 00010000:00010000:3.0:1787826440.670963:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826441.695003:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826442.719009:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826443.742963:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787826444.767298:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826445.023765:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564025984:192.168.203.139@tcp:400:kworker.0 00000100:00100000:2.0:1787826445.023917:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564026240:192.168.203.139@tcp:400:kworker.0 00000100:00100000:2.0:1787826445.023952:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:2.0:1787826445.023961:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:307626ff-a376-4faf-93e0-9f55b723d964:2404:1874670564025856:192.168.203.139@tcp:400:kworker.0 00000100:00100000:2.0:1787826445.024007:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564026112:192.168.203.139@tcp:400:kworker.0 00000100:00100000:2.0:1787826445.024028:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564026368:192.168.203.139@tcp:400:kworker.0 00010000:00010000:2.0:1787826445.791037:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1787826446.047247:0:42426:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826446.047263:0:42426:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826446.047288:0:42426:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826446.047301:0:42426:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826446.047305:0:42426:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826446.047312:0:42426:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826446.047322:0:42426:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x20:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826446.047326:0:42426:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826446.047331:0:42426:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x20:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826446.047337:0:42426:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x22:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826446.047341:0:42426:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826446.047345:0:42426:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x22:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826446.047351:0:42426:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x24:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826446.047354:0:42426:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826446.047359:0:42426:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x24:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826446.047375:0:42426:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x26:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826446.047379:0:42426:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826446.047384:0:42426:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x26:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:2.0:1787826446.559276:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 1 seconds 00010000:00010000:3.0:1787826446.815000:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826447.582985:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 1 seconds 00010000:00010000:2.0:1787826447.839055:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826448.607004:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 1 seconds 00010000:00010000:2.0:1787826448.863046:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826449.630942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 1 seconds 00010000:00010000:3.0:1787826449.886987:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826450.143056:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0161e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:307626ff-a376-4faf-93e0-9f55b723d964:2402:1874670564026496:192.168.203.139@tcp:400:kworker.0 00000100:00100000:0.0:1787826450.143240:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0163c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670564026752:192.168.203.139@tcp:400:kworker.0 00000100:00100000:1.0:1787826450.143252:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670564026624:192.168.203.139@tcp:400:kworker.0 00000100:00100000:0.0:1787826450.143274:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0162940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670564027008:192.168.203.139@tcp:400:kworker.0 00000100:00100000:1.0:1787826450.143296:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0161a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670564026880:192.168.203.139@tcp:400:kworker.0 00000100:00100000:2.0:1787826450.655078:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 1 seconds 00010000:00010000:2.0:1787826450.911020:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1787826451.167049:0:42426:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x28:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826451.167065:0:42426:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826451.167100:0:42426:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x28:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826451.167113:0:42426:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x2a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826451.167117:0:42426:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826451.167124:0:42426:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x2a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826451.167130:0:42426:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x2c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826451.167134:0:42426:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826451.167139:0:42426:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x2c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826451.167145:0:42426:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x2e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826451.167149:0:42426:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826451.167153:0:42426:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x2e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826451.167159:0:42426:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x30:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826451.167162:0:42426:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826451.167166:0:42426:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x30:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826451.167172:0:42426:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x32:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826451.167176:0:42426:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826451.167180:0:42426:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x32:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826451.167185:0:42426:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x34:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826451.167189:0:42426:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826451.167194:0:42426:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x34:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826451.167199:0:42426:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x36:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826451.167203:0:42426:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826451.167208:0:42426:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x36:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00000400:2.0:1787826451.678967:0:56419:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826435/real 1787826435] req@ffff976889cfb840 x1874670564024960/t0(0) o36->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 488/512 e 0 to 1 dl 1787826451 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'chown.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826451.688349:0:56419:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1787826451.688367:0:56419:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0001-mdc-ffff9768877a4800: Connection to lustre-MDT0001 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1787826451.693169:0:56419:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0001_UUID: waking up pinger s:DISCONN 00000100:00100000:2.0:1787826451.693201:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826451.693289:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826451.693296:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826451.693300:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826451.693310:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1123 00000100:00080000:1.0:1787826451.693323:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826451.693334:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826451.693353:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b01621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:307626ff-a376-4faf-93e0-9f55b723d964:2401:1874670564027136:192.168.203.139@tcp:400:kworker.0 00000100:00100000:1.0:1787826451.693358:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9768b0160f00 x1874670564027392/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826451.693431:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0160f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564027392:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826451.693444:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0163840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670564027264:192.168.203.139@tcp:400:kworker.0 00000100:00100000:0.0:1787826451.693504:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9768b0163480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670564027520:192.168.203.139@tcp:400:kworker.0 00000100:00100000:1.0:1787826451.693526:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670564027648:192.168.203.139@tcp:400:kworker.0 00010000:00010000:2.0:1787826451.935093:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826452.702967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826452.958961:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826453.727006:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826453.983046:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826454.750948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826455.007142:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826455.774938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00000400:1.0:1787826455.774949:0:2402:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826439/real 1787826439] req@ffff9768b0160780 x1874670564025344/t0(0) o400->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826455 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:0.0:1787826455.774950:0:2401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826439/real 1787826439] req@ffff9768b0162d00 x1874670564025216/t0(0) o400->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 224/224 e 0 to 1 dl 1787826455 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826455.775015:0:2401:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9768b0162d00 x1874670564025216/t0(0) o400->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 224/224 e 0 to 1 dl 1787826455 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826455.775027:0:2401:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from FULL to DISCONN 00000100:02020000:0.0:1787826455.775032:0:2401:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.203.139@tcp: Connection to MGS (at 192.168.203.139@tcp) was lost; in progress operations using this service will fail 00000100:00100000:1.0:1787826455.784986:0:2402:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9768b0160780 x1874670564025344/t0(0) o400->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826455 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826455.785000:0:2402:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:1.0:1787826455.785005:0:2402:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff9768877a4800: Connection to lustre-MDT0000 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:1.0:1787826455.785010:0:2402:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00000400:1.0:1787826455.785036:0:2402:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826439/real 1787826439] req@ffff9768b0162580 x1874670564025600/t0(0) o400->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826455 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826455.785052:0:2402:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9768b0162580 x1874670564025600/t0(0) o400->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826455 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826455.785059:0:2402:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:1.0:1787826455.785063:0:2402:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0000-osc-ffff9768877a4800: Connection to lustre-OST0000 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:1.0:1787826455.785068:0:2402:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0000_UUID: waking up pinger s:DISCONN 00000100:00100000:1.0:1787826455.785097:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0160780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670564025344:192.168.203.139@tcp:400:kworker.0 00000100:00100000:1.0:1787826455.785101:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0162580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670564025600:192.168.203.139@tcp:400:kworker.0 00000100:00080000:0.0:1787826455.793086:0:2401:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.203.139@tcp for MGC192.168.203.139@tcp not replayable, auto-deactivating 00000100:00080000:0.0:1787826455.793092:0:2401:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:0.0:1787826455.793101:0:2401:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff9768b0162d00 x1874670564025216/t0(0) o400->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 224/224 e 0 to 1 dl 1787826455 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826455.793121:0:2401:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff97688cb5cf00 x1874670564025856/t0(0) o400->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 224/224 e 0 to 0 dl 1787826461 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826455.793141:0:2401:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff9768b0161e00 x1874670564026496/t0(0) o400->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 224/224 e 0 to 0 dl 1787826466 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826455.793160:0:2401:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff9768b01621c0 x1874670564027136/t0(0) o400->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 224/224 e 0 to 0 dl 1787826467 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826455.793174:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 10000000:01000000:0.0:1787826455.793175:0:2401:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:0.0:1787826455.793178:0:2401:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:0.0:1787826455.793179:0:2401:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:2.0:1787826455.793179:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826455.793181:0:2401:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1787826455.793185:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826455.793191:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1282 00000100:00000400:0.0:1787826455.793200:0:2401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826439/real 1787826439] req@ffff9768b01603c0 x1874670564025472/t0(0) o400->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826455 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826455.793205:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787826455.793210:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:307626ff-a376-4faf-93e0-9f55b723d964:2404:1874670564025856:192.168.203.139@tcp:400:kworker.0 00000100:00100000:0.0:1787826455.793223:0:2401:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9768b01603c0 x1874670564025472/t0(0) o400->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826455 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826455.793231:0:2401:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff97688929c000 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:0.0:1787826455.793237:0:2401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826439/real 1787826439] req@ffff9768b0161680 x1874670564025728/t0(0) o400->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826455 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826455.793242:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564027776/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826455.793250:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0161e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:307626ff-a376-4faf-93e0-9f55b723d964:2402:1874670564026496:192.168.203.139@tcp:400:kworker.0 00000100:00100000:0.0:1787826455.793256:0:2401:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9768b0161680 x1874670564025728/t0(0) o400->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826455 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826455.793264:0:2401:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1787826455.793265:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:02000400:0.0:1787826455.793269:0:2401:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0001-osc-ffff9768877a4800: Connection to lustre-OST0001 (at 192.168.203.139@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1787826455.793269:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826455.793271:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826455.793274:0:2401:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0001_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1787826455.793275:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1282 00000100:00080000:2.0:1787826455.793281:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787826455.793291:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564027776:192.168.203.139@tcp:250:kworker.0 00000008:00100000:2.0:1787826455.793296:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826455.793302:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b01621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:307626ff-a376-4faf-93e0-9f55b723d964:2401:1874670564027136:192.168.203.139@tcp:400:kworker.0 00000100:00100000:0.0:1787826455.793309:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0162d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:307626ff-a376-4faf-93e0-9f55b723d964:2401:1874670564025216:192.168.203.139@tcp:400:kworker.0 00000100:00100000:2.0:1787826455.793310:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564027904/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826455.793315:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b01603c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670564025472:192.168.203.139@tcp:400:kworker.0 00000100:00100000:0.0:1787826455.793322:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0161680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670564025728:192.168.203.139@tcp:400:kworker.0 00000100:00080000:2.0:1787826455.793326:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826455.793329:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787826455.793331:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826455.793334:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826455.793335:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826455.793339:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1123 00000100:00080000:2.0:1787826455.793349:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826455.793354:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787826455.793373:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564028032/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826455.793381:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826455.793384:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826455.793386:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826455.793390:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1123 00000100:00080000:2.0:1787826455.793397:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826455.793402:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826455.793404:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564027904:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826455.793425:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564028160/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826455.793441:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787826455.793442:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564028032:192.168.203.139@tcp:8:kworker.0 00000100:00080000:2.0:1787826455.793447:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787826455.793452:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826455.793455:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1787826455.793491:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564028160:192.168.203.139@tcp:8:kworker.0 00010000:00010000:3.0:1787826456.031102:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1787826456.286934:0:36:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x38:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826456.286951:0:36:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826456.286993:0:36:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x38:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826456.287008:0:36:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x3a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826456.287013:0:36:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826456.287020:0:36:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x3a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826456.287026:0:36:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x3c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826456.287031:0:36:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826456.287037:0:36:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x3c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826456.287043:0:36:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x3e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826456.287047:0:36:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826456.287053:0:36:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x3e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826456.287059:0:36:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x40:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826456.287063:0:36:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826456.287068:0:36:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x40:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826456.287074:0:36:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x42:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826456.287078:0:36:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826456.287083:0:36:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x42:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:2.0:1787826456.799006:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787826456.799016:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826456.799023:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826456.799027:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826456.799031:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826456.799035:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826456.799038:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826456.799042:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826456.799045:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826456.799048:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826456.799051:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1787826457.055018:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826457.823030:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826458.078948:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826458.847051:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826459.103109:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826459.870985:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826460.127085:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826460.831667:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826460.831677:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787826460.831680:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826460.831683:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787826460.831686:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826460.831689:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787826460.831691:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826460.831694:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787826460.831697:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826460.831699:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1787826460.895011:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826461.151101:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:3.0:1787826461.215031:0:2404:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826445/real 1787826445] req@ffff97688cb5c3c0 x1874670564026112/t0(0) o400->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826461 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826461.241526:0:2404:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff97688cb5c3c0 x1874670564026112/t0(0) o400->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826461 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826461.241539:0:2404:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff97688929c000 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:3.0:1787826461.241547:0:2404:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826445/real 1787826445] req@ffff97688cb5f840 x1874670564026368/t0(0) o400->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826461 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826461.241560:0:2404:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff97688cb5f840 x1874670564026368/t0(0) o400->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826461 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826461.241566:0:2404:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff97688929b000 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:3.0:1787826461.241572:0:2404:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826445/real 1787826445] req@ffff97688cb5c780 x1874670564025984/t0(0) o400->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826461 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826461.241583:0:2404:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff97688cb5c780 x1874670564025984/t0(0) o400->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826461 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826461.241590:0:2404:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff976889e8e800 already not connected (conn 9, was 8): CONNECTING 00000100:00000400:3.0:1787826461.241595:0:2404:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826445/real 1787826445] req@ffff97688cb5da40 x1874670564026240/t0(0) o400->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826461 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826461.241605:0:2404:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff97688cb5da40 x1874670564026240/t0(0) o400->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826461 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826461.241609:0:2404:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff976889298000 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:3.0:1787826461.241623:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564026112:192.168.203.139@tcp:400:kworker.0 00000100:00100000:3.0:1787826461.241628:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564026368:192.168.203.139@tcp:400:kworker.0 00000100:00100000:3.0:1787826461.241631:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564025984:192.168.203.139@tcp:400:kworker.0 00000100:00100000:3.0:1787826461.241634:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2404:1874670564026240:192.168.203.139@tcp:400:kworker.0 00000080:00200000:0.0:1787826461.407004:0:36:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x44:0x0] ignore/verify layout 1/0, layout version 41 restore needed 0 00020000:00200000:0.0:1787826461.407022:0:36:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826461.407047:0:36:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x44:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:2.0:1787826461.918924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787826461.919083:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826451/real 1787826451] req@ffff9768b0160f00 x1874670564027392/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826461 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826461.919117:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9768b0160f00 x1874670564027392/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826461 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826461.919134:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826461.919152:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826461.919157:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0160f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564027392:192.168.203.139@tcp:38:kworker.0 00000100:00080000:1.0:1787826461.919190:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826461.919196:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787826461.919198:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826461.919201:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787826461.919204:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826461.919207:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826461.919211:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826461.919217:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1396 00000100:00080000:1.0:1787826461.919228:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1787826461.919235:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826461.919247:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826461.919280:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564028288/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826461.919306:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826461.919310:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787826461.919313:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826461.919316:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1787826461.919379:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564028288:192.168.203.139@tcp:38:kworker.0 00010000:00010000:3.0:1787826462.174980:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826462.942953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826463.199038:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826463.967078:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826464.223069:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826464.991665:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826465.246958:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1787826466.014974:0:2401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826450/real 1787826450] req@ffff9768b0160b40 x1874670564026624/t0(0) o400->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826466 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1787826466.014974:0:2402:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826450/real 1787826450] req@ffff9768b0163c00 x1874670564026752/t0(0) o400->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826466 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826466.015024:0:2402:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9768b0163c00 x1874670564026752/t0(0) o400->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826466 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826466.015034:0:2402:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff97688929c000 already not connected (conn 3, was 1): CONNECTING 00000100:00000400:1.0:1787826466.015041:0:2402:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826450/real 1787826450] req@ffff9768b0162940 x1874670564027008/t0(0) o400->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826466 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826466.015064:0:2402:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9768b0162940 x1874670564027008/t0(0) o400->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826466 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826466.015071:0:2402:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff97688929b000 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:1.0:1787826466.015096:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0163c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670564026752:192.168.203.139@tcp:400:kworker.0 00000100:00100000:1.0:1787826466.015104:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0162940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670564027008:192.168.203.139@tcp:400:kworker.0 00000100:00100000:2.0:1787826466.015126:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787826466.015152:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826455/real 1787826455] req@ffff976889cf8f00 x1874670564028160/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826465 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826466.015164:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564028160/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826465 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826466.015175:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826455/real 1787826455] req@ffff976889cf9e00 x1874670564027904/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826465 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826466.015187:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564027904/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826465 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826466.015197:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826455/real 1787826455] req@ffff976889cfa940 x1874670564028032/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826465 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826466.015217:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564028032/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826465 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826466.015228:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826455/real 1787826455] req@ffff976889cfbc00 x1874670564027776/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826465 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826466.015242:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564027776/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826465 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826466.015254:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826466.015268:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826466.015270:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564028160:192.168.203.139@tcp:8:kworker.0 00000100:00080000:1.0:1787826466.015274:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826466.015277:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826466.015279:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564027904:192.168.203.139@tcp:38:kworker.0 00000100:00080000:1.0:1787826466.015282:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826466.015285:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826466.015288:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564028032:192.168.203.139@tcp:8:kworker.0 00000100:00080000:1.0:1787826466.015292:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826466.015296:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826466.015298:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564027776:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787826466.015303:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826466.015309:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787826466.015312:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826466.015318:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1400 00000100:00080000:2.0:1787826466.015329:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1787826466.015351:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826466.015384:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564028416/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826466.015405:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826466.015411:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826466.015413:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826466.015417:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1400 00000100:00080000:2.0:1787826466.015422:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1787826466.015426:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826466.015436:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826466.015437:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564028416:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826466.015451:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564028544/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826466.015462:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826466.015465:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787826466.015468:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826466.015471:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826466.015481:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826466.015485:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1400 00000100:00080000:2.0:1787826466.015489:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1787826466.015493:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826466.015497:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787826466.015507:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9a40 x1874670564028672/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826466.015515:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826466.015519:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826466.015521:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826466.015524:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1400 00000100:00080000:2.0:1787826466.015527:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1787826466.015529:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826466.015533:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787826466.015556:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564028800/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826466.015718:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564028544:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826466.015747:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564028672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787826466.015769:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564028800:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826466.029411:0:2401:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9768b0160b40 x1874670564026624/t0(0) o400->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826466 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826466.029425:0:2401:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff976889e8e800 already not connected (conn 10, was 8): CONNECTING 00000100:00000400:0.0:1787826466.029433:0:2401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826450/real 1787826450] req@ffff9768b0161a40 x1874670564026880/t0(0) o400->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826466 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826466.029447:0:2401:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9768b0161a40 x1874670564026880/t0(0) o400->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826466 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826466.029454:0:2401:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff976889298000 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:0.0:1787826466.029469:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0160b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670564026624:192.168.203.139@tcp:400:kworker.0 00000100:00100000:0.0:1787826466.029475:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0161a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670564026880:192.168.203.139@tcp:400:kworker.0 00010000:00010000:2.0:1787826466.270981:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1787826466.527274:0:42426:0:(vvp_io.c:1889:vvp_io_init()) [0x240000404:0x14:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826466.527290:0:42426:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:3.0:1787826466.527299:0:42426:0:(lov_io.c:1253:lov_io_iter_init()) component[1] flags 0x0 00000008:00010000:3.0:1787826466.527317:0:42426:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000005e3980cb ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab1943 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020080000000 nid: local remote: 0x22153cd396b83ce1 expref: -99 pid: 56198 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787826466.527348:0:42426:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000005e3980cb ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab1943 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020080000000 nid: local remote: 0x22153cd396b83ce1 expref: -99 pid: 56198 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826466.527400:0:42426:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab1943 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020080000000 nid: local remote: 0x22153cd396b83ce1 expref: -99 pid: 56198 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:3.0:1787826466.527506:0:42426:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff97688cb5da40 - object 0x280000400:40 offset 0<>4 00000080:00200000:3.0:1787826466.527549:0:42426:0:(vvp_io.c:304:vvp_io_fini()) [0x240000404:0x14:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00080000:2.0:1787826466.527815:0:2404:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff97688cb5da40 x1874670564028928/t0(0) o4->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:6/4 lens 496/448 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1787826467.038927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00000400:0.0:1787826467.038936:0:2401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826451/real 1787826451] req@ffff9768b0163480 x1874670564027520/t0(0) o400->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826467 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826467.038968:0:2401:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9768b0163480 x1874670564027520/t0(0) o400->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826467 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826467.038976:0:2401:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff976889298000 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:0.0:1787826467.038989:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0163480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2401:1874670564027520:192.168.203.139@tcp:400:kworker.0 00000100:00000400:1.0:1787826467.039003:0:2402:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826451/real 1787826451] req@ffff9768b0163840 x1874670564027264/t0(0) o400->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826467 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826467.039022:0:2402:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9768b0163840 x1874670564027264/t0(0) o400->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826467 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826467.039030:0:2402:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff976889e8e800 already not connected (conn 10, was 8): CONNECTING 00000100:00000400:1.0:1787826467.039037:0:2402:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826451/real 1787826451] req@ffff976889efd680 x1874670564027648/t0(0) o400->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826467 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826467.039048:0:2402:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff976889efd680 x1874670564027648/t0(0) o400->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826467 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826467.039054:0:2402:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff97688929b000 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:1.0:1787826467.039062:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9768b0163840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670564027264:192.168.203.139@tcp:400:kworker.0 00000100:00100000:1.0:1787826467.039067:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2402:1874670564027648:192.168.203.139@tcp:400:kworker.0 00010000:00010000:2.0:1787826467.295034:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826468.063179:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826468.318967:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826469.086957:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826469.343358:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826470.111008:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826470.367026:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787826471.135303:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826471.135314:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787826471.135318:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826471.135322:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787826471.135325:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826471.135328:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787826471.135331:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826471.135334:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787826471.135336:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826471.135340:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1787826471.136116:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826471.391013:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826472.158954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826472.415363:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826473.182963:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826473.439010:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826474.206958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826474.463074:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826475.230949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826475.487036:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826476.254916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826476.254925:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826461/real 1787826461] req@ffff976889efc3c0 x1874670564028288/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826476 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826476.254959:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564028288/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826476 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826476.254987:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826476.255010:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826476.255014:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564028288:192.168.203.139@tcp:38:kworker.0 00000100:00080000:0.0:1787826476.255073:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826476.255077:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787826476.255079:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826476.255081:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787826476.255083:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826476.255086:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826476.255089:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826476.255094:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1406 00000100:00080000:0.0:1787826476.255104:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1787826476.255110:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826476.255122:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826476.255152:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564029056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826476.255174:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826476.255177:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787826476.255180:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826476.255183:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1787826476.255253:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564029056:192.168.203.139@tcp:38:kworker.0 00010000:00010000:1.0:1787826476.511009:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826477.279014:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826477.534998:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826478.302918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826478.559050:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826479.326906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826479.583034:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826480.351467:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826480.606967:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826481.374898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826481.374945:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826481.374955:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787826481.374958:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826481.374961:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787826481.374964:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826481.374967:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787826481.374970:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826481.374974:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787826481.374976:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826481.374979:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1787826481.375056:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826466/real 1787826466] req@ffff976889cfa1c0 x1874670564028544/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826481 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826481.375083:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564028544/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826481 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826481.375093:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826466/real 1787826466] req@ffff976889cf9a40 x1874670564028672/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826481 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826481.375103:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9a40 x1874670564028672/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826481 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826481.375111:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826466/real 1787826466] req@ffff976889cfb0c0 x1874670564028800/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826481 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826481.375121:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564028800/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826481 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826481.375130:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826466/real 1787826466] req@ffff976889cf8000 x1874670564028416/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826481 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826481.375140:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564028416/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826481 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826481.375159:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826481.375211:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826481.375216:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564028544:192.168.203.139@tcp:38:kworker.0 00000100:00080000:2.0:1787826481.375222:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826481.375267:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826481.375271:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787826481.375273:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826481.375276:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826481.375278:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826481.375283:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1411 00000100:00080000:1.0:1787826481.375293:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1787826481.375299:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826481.375310:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826481.375339:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd680 x1874670564029184/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826481.375351:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826481.375354:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787826481.375356:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826481.375359:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826481.375361:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826481.375364:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1411 00000100:00080000:1.0:1787826481.375367:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1787826481.375371:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826481.375375:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826481.375384:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564029312/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826481.375392:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826481.375394:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787826481.375435:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826481.375442:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564028672:192.168.203.139@tcp:8:kworker.0 00000100:00080000:2.0:1787826481.375447:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826481.375452:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826481.375456:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564028800:192.168.203.139@tcp:8:kworker.0 00000100:00080000:2.0:1787826481.375460:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826481.375465:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826481.375469:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564028416:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787826481.375472:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826481.375477:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826481.375480:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826481.375488:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1411 00000100:00080000:1.0:1787826481.375495:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1787826481.375501:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826481.375516:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564029184:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826481.375519:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564029440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826481.375533:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826481.375536:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787826481.375539:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826481.375542:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826481.375544:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826481.375548:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1411 00000100:00080000:1.0:1787826481.375552:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1787826481.375556:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826481.375563:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787826481.375576:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564029568/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826481.375864:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564029312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826481.375897:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564029440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826481.375975:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564029568:192.168.203.139@tcp:8:kworker.0 00010000:00010000:2.0:1787826481.630992:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826482.398926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826482.655161:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826483.422928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826483.679029:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826484.446944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826484.703173:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826485.470968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826485.728551:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787826486.494967:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787826486.494968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826486.494981:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787826486.494986:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826486.494989:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787826486.494992:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826486.494995:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787826486.494997:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826486.495001:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787826486.495003:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826486.495006:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1787826486.751067:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826487.519509:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826487.775006:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826488.542942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826488.799035:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826489.568466:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826489.822961:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826490.591236:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826490.847100:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787826491.616083:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826491.616093:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787826491.616095:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826491.616099:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787826491.616103:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826491.616106:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787826491.616109:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826491.616112:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787826491.616116:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826491.616119:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1787826491.616336:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826491.871369:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826492.638988:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826492.894970:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:3.0:1787826493.342950:0:2392:0:(socklnd_cb.c:2254:ksocknal_find_timed_out_conn()) A connection with 12345-192.168.203.139@tcp (192.168.203.139:988) timed out; the network or node may be down. 00000100:00100000:0.0:1787826493.663364:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826493.919018:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826494.686920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826494.945409:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826495.711258:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826495.967144:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826496.734978:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787826496.735003:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787826496.735007:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826476/real 1787826476] req@ffff976882aed680 x1874670564029056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826496 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826496.735011:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787826496.735016:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826496.735033:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787826496.735041:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826496.735051:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787826496.735063:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826496.735075:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787826496.735079:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826496.735081:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1787826496.735114:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564029056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826496 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826496.735210:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826496.735235:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826496.735243:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564029056:192.168.203.139@tcp:38:kworker.0 00000100:00080000:1.0:1787826496.735243:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826496.735261:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787826496.735264:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826496.735268:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787826496.735271:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826496.735274:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826496.735278:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826496.735289:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1421 00000100:00080000:1.0:1787826496.735302:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 15s 00000100:00080000:1.0:1787826496.735310:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826496.735322:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826496.735356:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564029696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826496.735389:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826496.735393:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787826496.735396:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826496.735400:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1787826496.735442:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564029696:192.168.203.139@tcp:38:kworker.0 00010000:00010000:2.0:1787826496.991039:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826497.759174:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826498.016005:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787826498.464782:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.203.139:988 00000400:00000100:3.0:1787826498.464860:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787826498.464880:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:2.0:1787826498.465237:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826496/real 1787826498] req@ffff976889efed00 x1874670564029696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826498.465261:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564029696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826498.465273:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826498.465277:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826498.465343:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564029696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787826498.782991:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826499.038939:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826499.806991:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826500.063022:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826500.830968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826501.087140:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826501.854936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826501.855049:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826501.855057:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787826501.855062:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826501.855066:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787826501.855069:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826501.855072:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826501.855077:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826501.855096:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1441 00000100:00080000:1.0:1787826501.855110:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826501.855120:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826501.855137:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826501.855326:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826481/real 1787826481] req@ffff976889effc00 x1874670564029440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826501 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826501.855361:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564029440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826501 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826501.855380:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564029824/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826501.855381:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826481/real 1787826481] req@ffff976889eff0c0 x1874670564029568/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826501 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826501.855439:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564029568/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826501 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826501.855450:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826481/real 1787826481] req@ffff976889efd680 x1874670564029184/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826501 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826501.855453:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826501.855466:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787826501.855473:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787826501.855478:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd680 x1874670564029184/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826501 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826501.855491:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1787826501.855498:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826481/real 1787826481] req@ffff976889efe940 x1874670564029312/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826501 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826501.855510:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564029312/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826501 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826501.855546:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564029824:192.168.203.139@tcp:38:kworker.0 00000100:00080000:0.0:1787826501.855680:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826501.855711:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826501.855716:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564029440:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787826501.855722:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826501.855727:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826501.855729:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564029568:192.168.203.139@tcp:8:kworker.0 00000100:00080000:0.0:1787826501.855735:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826501.855740:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826501.855742:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564029184:192.168.203.139@tcp:38:kworker.0 00000100:00080000:0.0:1787826501.855746:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826501.855750:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826501.855752:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564029312:192.168.203.139@tcp:8:kworker.0 00000100:00080000:2.0:1787826501.856526:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826501.856532:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787826501.856535:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826501.856542:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1426 00000100:00080000:2.0:1787826501.856550:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1787826501.856557:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826501.856590:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564029952/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826501.856606:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826501.856611:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826501.856613:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826501.856617:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1426 00000100:00080000:2.0:1787826501.856621:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1787826501.856627:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826501.856638:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826501.856663:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564030080/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826501.856674:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826501.856677:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826501.856679:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826501.856685:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1426 00000100:00080000:2.0:1787826501.856690:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1787826501.856695:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826501.856699:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787826501.856710:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9a40 x1874670564030208/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826501.856719:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826501.856721:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826501.856723:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826501.856726:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1426 00000100:00080000:2.0:1787826501.856729:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1787826501.856732:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826501.856737:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787826501.856745:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564030336/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826501.856930:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564029952:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826501.856972:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564030080:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826501.856986:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564030208:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826501.856997:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564030336:192.168.203.139@tcp:8:kworker.0 00010000:00010000:3.0:1787826502.110978:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826502.878909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826503.136697:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826503.902951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826504.159022:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787826504.607580:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.203.139:988 00000400:00000100:1.0:1787826504.607628:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787826504.607657:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787826504.607707:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787826504.607721:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787826504.607734:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787826504.607750:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:0.0:1787826504.607869:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826501/real 1787826504] req@ffff976889cf8000 x1874670564029952/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826504.607914:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564029952/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826504.607927:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826504.607932:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826504.607938:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564029952:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826504.607948:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826501/real 1787826504] req@ffff976889cfb0c0 x1874670564030080/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826504.607962:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564030080/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826504.607971:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826504.607975:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826504.607981:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564030080:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826504.607995:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826501/real 1787826504] req@ffff976889cf9a40 x1874670564030208/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826504.608007:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9a40 x1874670564030208/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826504.608016:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826504.608019:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826504.608022:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564030208:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826504.608029:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826501/real 1787826504] req@ffff976889cfa1c0 x1874670564030336/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826504.608041:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564030336/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826504.608048:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826504.608051:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826504.608054:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564030336:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826504.608061:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826501/real 1787826504] req@ffff976889eff480 x1874670564029824/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826504.608072:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564029824/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826504.608079:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826504.608082:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826504.608085:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564029824:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787826504.926899:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826505.183085:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826505.950882:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826506.206981:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826506.974985:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787826506.974999:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826506.975014:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787826506.975022:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826506.975038:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1446 00000100:00080000:2.0:1787826506.975053:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826506.975061:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826506.975113:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564030464/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826506.975145:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826506.975148:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826506.975150:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826506.975154:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1446 00000100:00080000:2.0:1787826506.975157:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826506.975169:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826506.975267:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826506.975283:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564030592/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826506.975292:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826506.975308:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787826506.975310:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826506.975318:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1446 00000100:00080000:2.0:1787826506.975326:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826506.975331:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826506.975336:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826506.975349:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564030720/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826506.975359:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564030464:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787826506.975391:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826506.975394:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826506.975397:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826506.975403:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1446 00000100:00080000:2.0:1787826506.975406:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826506.975411:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826506.975419:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787826506.975434:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564030848/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826506.975442:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826506.975447:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826506.975449:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826506.975462:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1446 00000100:00080000:2.0:1787826506.975476:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826506.975482:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826506.975487:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787826506.975500:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564030976/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826506.975520:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564030592:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826506.975583:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564030720:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826506.975611:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564030848:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826506.975654:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564030976:192.168.203.139@tcp:8:kworker.0 00010000:00010000:1.0:1787826507.231003:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787826507.679137:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.203.139:988 00000400:00000100:3.0:1787826507.679176:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787826507.679198:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826507.679237:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826507.679248:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826507.679258:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826507.679271:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:0.0:1787826507.679283:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826506/real 1787826507] req@ffff976889cfad00 x1874670564030720/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826507.679313:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564030720/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826507.679325:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826507.679330:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826507.679339:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564030720:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826507.679347:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826506/real 1787826507] req@ffff976889cf92c0 x1874670564030848/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826507.679359:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564030848/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826507.679367:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826507.679370:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826507.679373:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564030848:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826507.679380:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826506/real 1787826507] req@ffff976889cfa1c0 x1874670564030976/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826507.679391:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564030976/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826507.679398:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826507.679401:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826507.679404:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564030976:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826507.679409:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826506/real 1787826507] req@ffff976889cfa580 x1874670564030464/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826507.679422:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564030464/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826507.679428:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826507.679431:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826507.679434:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564030464:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826507.679439:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826506/real 1787826507] req@ffff976889cfb480 x1874670564030592/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826507.679449:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564030592/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826507.679455:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826507.679459:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826507.679462:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564030592:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787826507.998938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826508.254996:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826509.022941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826509.279102:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826510.046951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826510.303162:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826511.071139:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826511.327027:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787826512.095014:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787826512.095015:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826512.095030:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826512.095035:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826512.095045:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1451 00000100:00080000:1.0:1787826512.095067:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826512.095076:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826512.095112:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564031104/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826512.095136:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826512.095139:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826512.095140:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826512.095144:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1451 00000100:00080000:1.0:1787826512.095148:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826512.095152:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826512.095163:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826512.095177:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564031232/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826512.095181:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564031104:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787826512.095206:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826512.095216:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826512.095218:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826512.095223:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1451 00000100:00080000:1.0:1787826512.095227:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826512.095230:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826512.095236:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826512.095250:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564031360/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826512.095277:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826512.095281:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826512.095282:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826512.095287:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1451 00000100:00080000:1.0:1787826512.095292:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826512.095296:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826512.095302:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826512.095315:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd2c0 x1874670564031488/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826512.095325:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826512.095328:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826512.095330:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826512.095333:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1451 00000100:00080000:1.0:1787826512.095335:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826512.095338:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826512.095341:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787826512.095355:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564031616/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826512.095387:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564031232:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826512.095427:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564031360:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826512.095455:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564031488:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826512.095487:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564031616:192.168.203.139@tcp:8:kworker.0 00010000:00010000:2.0:1787826512.351099:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826513.118966:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826513.375097:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787826513.823304:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.203.139:988 00000400:00000100:3.0:1787826513.823350:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787826513.823365:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826513.823404:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826513.823415:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826513.823422:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826513.823429:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:0.0:1787826513.823493:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826512/real 1787826513] req@ffff976889efe580 x1874670564031232/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826513.823516:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564031232/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826513.823528:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826513.823533:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826513.823537:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564031232:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826513.823545:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826512/real 1787826513] req@ffff976889efcb40 x1874670564031360/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826513.823583:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564031360/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826513.823590:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826513.823594:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826513.823597:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564031360:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826513.823603:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826512/real 1787826513] req@ffff976889efd2c0 x1874670564031488/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826513.823614:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd2c0 x1874670564031488/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826513.823621:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826513.823625:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826513.823628:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564031488:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826513.823634:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826512/real 1787826513] req@ffff976889efc780 x1874670564031616/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826513.823644:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564031616/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826513.823655:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826513.823658:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826513.823661:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564031616:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826513.823666:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826512/real 1787826513] req@ffff976889efe1c0 x1874670564031104/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826513.823676:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564031104/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826513.823683:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826513.823686:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826513.823689:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564031104:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826514.142966:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826514.398958:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826515.166950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826515.423544:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826516.190927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826516.447535:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826517.214941:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787826517.214945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787826517.214953:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826517.214958:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826517.214967:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1457 00000100:00080000:0.0:1787826517.214978:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826517.214985:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787826517.215020:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564031744/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826517.215045:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826517.215049:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826517.215051:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826517.215055:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1457 00000100:00080000:0.0:1787826517.215058:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826517.215062:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826517.215071:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826517.215082:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef840 x1874670564031872/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826517.215092:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787826517.215095:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564031744:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787826517.215096:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826517.215100:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826517.215106:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1457 00000100:00080000:0.0:1787826517.215109:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826517.215113:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826517.215117:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826517.215128:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564032000/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826517.215137:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826517.215141:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826517.215143:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826517.215147:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1457 00000100:00080000:0.0:1787826517.215151:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826517.215166:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:2.0:1787826517.215172:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:0.0:1787826517.215173:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787826517.215188:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564032128/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826517.215197:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826517.215201:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826517.215202:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826517.215206:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1457 00000100:00080000:0.0:1787826517.215209:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826517.215213:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826517.215217:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787826517.215221:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826517/real 1787826517] req@ffff976882aed680 x1874670564031744/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826517.215230:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564032256/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826517.215241:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564031744/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826517.215263:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826517.215267:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826517.215271:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564031744:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826517.215291:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564031872:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826517.215312:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826517.215324:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826517/real 1787826517] req@ffff976882aef840 x1874670564031872/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826517.215335:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef840 x1874670564031872/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826517.215343:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826517.215346:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826517.215350:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564031872:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826517.215356:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564032000:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826517.215373:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826517.215383:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826517/real 1787826517] req@ffff976882aeed00 x1874670564032000/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826517.215404:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564032000/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826517.215412:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826517.215414:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826517.215417:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564032000:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826517.215421:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564032128:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826517.215439:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826517.215448:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826517/real 1787826517] req@ffff976882aee580 x1874670564032128/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826517.215458:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564032128/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826517.215465:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826517.215468:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826517.215479:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564032128:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826517.215488:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564032256:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826517.215506:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826517.215516:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826517/real 1787826517] req@ffff976882aee940 x1874670564032256/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826517.215526:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564032256/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826517.215538:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826517.215542:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826517.215556:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564032256:192.168.203.139@tcp:8:kworker.0 00010000:00010000:3.0:1787826517.473920:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826518.238947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826518.494981:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826519.262985:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826519.519096:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826520.286992:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826520.542973:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826521.310995:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826521.567018:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826522.334899:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826522.334908:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826522.334912:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787826522.334912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787826522.334924:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1462 00000100:00080000:0.0:1787826522.334935:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826522.334942:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787826522.334979:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564032384/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826522.335010:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826522.335014:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826522.335015:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826522.335019:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1462 00000100:00080000:0.0:1787826522.335022:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826522.335026:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826522.335036:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826522.335117:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564032384:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826522.335247:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564032512/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826522.335265:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826522.335268:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826522.335270:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826522.335273:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1462 00000100:00080000:0.0:1787826522.335277:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826522.335281:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826522.335287:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826522.335300:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564032640/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826522.335308:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826522.335311:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826522.335312:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826522.335315:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1462 00000100:00080000:0.0:1787826522.335318:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826522.335321:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826522.335324:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787826522.335331:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564032768/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826522.335336:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826522.335338:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826522.335339:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826522.335340:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1462 00000100:00080000:0.0:1787826522.335342:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826522.335344:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826522.335346:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826522.335358:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564032896/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826522.335416:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564032512:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826522.335447:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564032640:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826522.335483:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564032768:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826522.335530:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564032896:192.168.203.139@tcp:8:kworker.0 00010000:00010000:2.0:1787826522.591015:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787826523.039106:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.203.139:988 00000400:00000100:3.0:1787826523.039144:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787826523.039159:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826523.039200:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826523.039208:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826523.039215:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826523.039223:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:2.0:1787826523.039314:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826522/real 1787826523] req@ffff976882aef0c0 x1874670564032512/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826523.039336:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564032512/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826523.039347:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826523.039351:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826523.039355:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564032512:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826523.039362:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826522/real 1787826523] req@ffff976882aee940 x1874670564032640/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826523.039374:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564032640/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826523.039381:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826523.039384:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826523.039387:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564032640:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826523.039393:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826522/real 1787826523] req@ffff976882aee580 x1874670564032768/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826523.039402:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564032768/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826523.039409:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826523.039412:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826523.039415:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564032768:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826523.039421:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826522/real 1787826523] req@ffff976882aeed00 x1874670564032896/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826523.039431:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564032896/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826523.039438:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826523.039441:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826523.039444:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564032896:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826523.039449:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826522/real 1787826523] req@ffff976882aec3c0 x1874670564032384/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826523.039461:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564032384/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826523.039468:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826523.039470:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826523.039473:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564032384:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826523.359142:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826523.614962:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826524.382906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826524.639198:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826525.407035:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826525.663537:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826526.431254:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826526.686986:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826527.454926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826527.455150:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826527.455156:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826527.455169:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826527.455178:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1467 00000100:00080000:1.0:1787826527.455188:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826527.455196:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826527.455229:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efda40 x1874670564033024/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826527.455255:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826527.455258:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826527.455260:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826527.455267:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1467 00000100:00080000:1.0:1787826527.455270:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826527.455274:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826527.455284:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826527.455295:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564033024:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826527.455299:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564033152/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826527.455310:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826527.455313:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826527.455314:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826527.455318:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1467 00000100:00080000:1.0:1787826527.455321:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826527.455324:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826527.455328:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826527.455338:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564033280/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826527.455346:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826527.455350:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826527.455351:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826527.455354:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1467 00000100:00080000:1.0:1787826527.455358:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787826527.455359:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787826527.455365:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826527.455370:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826527.455393:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd2c0 x1874670564033408/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826527.455403:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787826527.455403:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826527/real 1787826527] req@ffff976889efda40 x1874670564033024/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826527.455411:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826527.455414:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826527.455422:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1467 00000100:00080000:1.0:1787826527.455426:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787826527.455430:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efda40 x1874670564033024/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826527.455432:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826527.455440:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787826527.455446:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826527.455453:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826527.455457:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564033024:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826527.455465:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564033536/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826527.455494:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564033152:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826527.455518:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826527.455531:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826527/real 1787826527] req@ffff976889efe1c0 x1874670564033152/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826527.455544:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564033152/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826527.455553:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826527.455556:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826527.455559:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564033152:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826527.455565:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564033280:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826527.455584:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826527.455594:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826527/real 1787826527] req@ffff976889efc780 x1874670564033280/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826527.455604:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564033280/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826527.455611:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826527.455614:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826527.455616:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564033280:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826527.455622:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564033408:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826527.455640:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826527.455651:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826527/real 1787826527] req@ffff976889efd2c0 x1874670564033408/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826527.455661:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd2c0 x1874670564033408/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826527.455669:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826527.455672:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826527.455675:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564033408:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826527.455680:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564033536:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826527.455698:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826527.455708:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826527/real 1787826527] req@ffff976889efcb40 x1874670564033536/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826527.455719:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564033536/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826527.455727:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826527.455730:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826527.455733:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564033536:192.168.203.139@tcp:8:kworker.0 00010000:00010000:2.0:1787826527.711129:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826528.478981:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826528.735040:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826529.503006:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826529.758963:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826530.526950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826530.782960:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826531.550954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826531.807000:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787826532.574943:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787826532.574947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826532.574959:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826532.574964:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826532.574973:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1472 00000100:00080000:1.0:1787826532.574984:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826532.574992:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826532.575031:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564033664/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826532.575058:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826532.575061:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826532.575063:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826532.575067:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1472 00000100:00080000:1.0:1787826532.575070:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826532.575074:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826532.575083:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826532.575094:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564033792/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826532.575102:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826532.575105:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826532.575107:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826532.575110:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1472 00000100:00080000:1.0:1787826532.575113:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826532.575116:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826532.575120:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826532.575130:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564033920/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826532.575138:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787826532.575142:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564033664:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787826532.575143:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826532.575146:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826532.575154:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1472 00000100:00080000:1.0:1787826532.575157:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826532.575161:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826532.575166:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826532.575177:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd680 x1874670564034048/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826532.575185:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826532.575188:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826532.575190:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826532.575194:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1472 00000100:00080000:1.0:1787826532.575198:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826532.575201:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826532.575204:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787826532.575215:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564034176/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787826532.575222:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826532.575286:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826532/real 1787826532] req@ffff976889efe580 x1874670564033664/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826532.575301:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564033664/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826532.575311:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826532.575315:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826532.575319:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564033664:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826532.575325:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564033792:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826532.575344:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826532.575355:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826532/real 1787826532] req@ffff976889eff480 x1874670564033792/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826532.575365:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564033792/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826532.575372:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826532.575375:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826532.575378:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564033792:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826532.575404:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564033920:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826532.575422:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826532.575432:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826532/real 1787826532] req@ffff976889efe940 x1874670564033920/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826532.575459:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564033920/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826532.575472:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826532.575475:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826532.575478:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564033920:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826532.575484:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564034048:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826532.575503:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826532.575514:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826532/real 1787826532] req@ffff976889efd680 x1874670564034048/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826532.575525:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd680 x1874670564034048/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826532.575531:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826532.575534:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826532.575537:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564034048:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826532.575541:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564034176:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826532.575554:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826532.575564:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826532/real 1787826532] req@ffff976889eff0c0 x1874670564034176/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826532.575574:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564034176/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826532.575581:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826532.575584:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826532.575587:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564034176:192.168.203.139@tcp:8:kworker.0 00010000:00010000:3.0:1787826532.831025:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826533.598935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826533.854982:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826534.622984:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826534.879053:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826535.647002:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826535.903569:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826536.670929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826536.928489:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826537.694936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826537.695026:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826537.695030:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826537.695033:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826537.695041:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1477 00000100:00080000:1.0:1787826537.695049:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826537.695055:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826537.695085:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564034304/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826537.695105:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826537.695109:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826537.695110:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826537.695114:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1477 00000100:00080000:1.0:1787826537.695116:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826537.695118:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826537.695125:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826537.695132:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564034432/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826537.695137:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826537.695139:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826537.695140:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826537.695142:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1477 00000100:00080000:1.0:1787826537.695143:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826537.695145:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826537.695147:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826537.695152:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564034560/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826537.695157:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826537.695158:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826537.695159:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826537.695161:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1477 00000100:00080000:1.0:1787826537.695163:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826537.695165:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826537.695168:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826537.695175:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564034688/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826537.695180:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826537.695182:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826537.695182:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826537.695184:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1477 00000100:00080000:1.0:1787826537.695186:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826537.695187:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826537.695189:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787826537.695196:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd680 x1874670564034816/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826537.695428:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564034304:192.168.203.139@tcp:250:kworker.0 00000800:00000100:2.0:1787826537.695477:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826537.695510:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826537/real 1787826537] req@ffff976889effc00 x1874670564034304/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826537.695521:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564034304/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826537.695528:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826537.695531:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826537.695534:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564034304:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826537.695540:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564034432:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826537.695552:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826537.695560:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826537/real 1787826537] req@ffff976889efed00 x1874670564034432/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826537.695572:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564034432/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826537.695585:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826537.695588:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826537.695591:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564034432:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826537.695597:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564034560:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826537.695614:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826537.695625:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826537/real 1787826537] req@ffff976889efc3c0 x1874670564034560/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826537.695636:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564034560/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826537.695643:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826537.695646:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826537.695649:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564034560:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826537.695655:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564034688:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826537.695675:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826537.695684:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826537/real 1787826537] req@ffff976889eff0c0 x1874670564034688/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826537.695692:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564034688/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826537.695697:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826537.695699:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826537.695701:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564034688:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826537.695706:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564034816:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826537.695719:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826537.695725:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826537/real 1787826537] req@ffff976889efd680 x1874670564034816/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826537.695735:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd680 x1874670564034816/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826537.695741:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826537.695743:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826537.695745:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564034816:192.168.203.139@tcp:8:kworker.0 00010000:00010000:2.0:1787826537.951062:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826538.718957:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826538.975186:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826539.742958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826539.999175:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826540.766946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826541.024769:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826541.791492:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826542.046965:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826542.814944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787826542.815193:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826542.815198:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826542.815202:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826542.815217:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1482 00000100:00080000:3.0:1787826542.815228:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826542.815236:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787826542.815272:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564034944/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826542.815299:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826542.815302:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826542.815304:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826542.815307:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1482 00000100:00080000:3.0:1787826542.815310:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826542.815313:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826542.815320:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826542.815327:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564035072/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826542.815333:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826542.815335:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826542.815336:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826542.815338:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1482 00000100:00080000:3.0:1787826542.815339:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826542.815341:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826542.815344:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826542.815348:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564035200/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826542.815354:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826542.815355:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826542.815356:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826542.815358:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1482 00000100:00080000:3.0:1787826542.815360:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826542.815363:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826542.815376:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787826542.815390:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564035328/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826542.815401:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826542.815404:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826542.815406:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787826542.815409:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564034944:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787826542.815413:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1482 00000100:00080000:3.0:1787826542.815419:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826542.815422:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826542.815427:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826542.815439:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564035456/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826542.815508:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564035072:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826542.815542:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564035200:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826542.815576:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564035328:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826542.815602:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564035456:192.168.203.139@tcp:8:kworker.0 00010000:00010000:2.0:1787826543.071109:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826543.838932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826544.095105:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787826544.543179:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.203.139:988 00000400:00000100:0.0:1787826544.543215:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787826544.543229:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:2.0:1787826544.543304:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826542/real 1787826544] req@ffff97688cb5c780 x1874670564034944/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826544.543328:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564034944/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826544.543340:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826544.543344:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000800:00000100:0.0:1787826544.543350:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:2.0:1787826544.543352:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564034944:192.168.203.139@tcp:250:kworker.0 00000800:00000100:0.0:1787826544.543379:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:2.0:1787826544.543379:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826542/real 1787826544] req@ffff97688cb5f840 x1874670564035072/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787826544.543391:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:2.0:1787826544.543399:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564035072/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787826544.543405:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00080000:2.0:1787826544.543415:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826544.543420:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826544.543424:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564035072:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826544.543430:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826542/real 1787826544] req@ffff97688cb5c3c0 x1874670564035200/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826544.543442:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564035200/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826544.543449:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826544.543453:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826544.543456:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564035200:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826544.543483:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826542/real 1787826544] req@ffff97688cb5cf00 x1874670564035328/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826544.543495:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564035328/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826544.543502:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826544.543505:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826544.543508:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564035328:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826544.543514:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826542/real 1787826544] req@ffff97688cb5cb40 x1874670564035456/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826544.543524:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564035456/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826544.543531:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826544.543535:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826544.543538:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564035456:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826544.862942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826545.118924:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826545.886942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826546.142984:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826546.911032:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826547.167365:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826547.934943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787826547.934985:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826547.934996:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826547.935001:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826547.935014:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1487 00000100:00080000:0.0:1787826547.935029:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826547.935038:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787826547.935087:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef840 x1874670564035584/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826547.935124:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826547.935128:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826547.935130:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826547.935134:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1487 00000100:00080000:0.0:1787826547.935138:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826547.935143:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826547.935155:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826547.935170:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564035712/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826547.935180:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826547.935184:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826547.935186:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787826547.935189:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564035584:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787826547.935193:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1487 00000100:00080000:0.0:1787826547.935200:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826547.935204:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826547.935209:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826547.935223:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564035840/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826547.935234:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826547.935236:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826547.935238:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826547.935241:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1487 00000100:00080000:0.0:1787826547.935245:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826547.935261:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826547.935267:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1787826547.935276:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826547.935285:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564035968/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826547.935296:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826547.935299:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826547.935301:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826547.935305:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1487 00000100:00080000:0.0:1787826547.935308:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826547.935312:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826547.935316:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826547.935330:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564036096/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826547.935420:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826547/real 1787826547] req@ffff976882aef840 x1874670564035584/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826547.935441:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef840 x1874670564035584/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826547.935452:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826547.935456:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826547.935460:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564035584:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826547.935484:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564035712:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826547.935519:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826547.935545:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826547/real 1787826547] req@ffff976882aed680 x1874670564035712/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826547.935559:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564035712/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826547.935567:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826547.935571:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826547.935574:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564035712:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826547.935580:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564035840:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826547.935599:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826547.935609:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826547/real 1787826547] req@ffff976882aec3c0 x1874670564035840/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826547.935626:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564035840/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826547.935633:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826547.935636:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826547.935638:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564035840:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826547.935656:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564035968:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826547.935676:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826547.935686:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826547/real 1787826547] req@ffff976882aeed00 x1874670564035968/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826547.935696:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564035968/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826547.935704:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826547.935706:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826547.935709:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564035968:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826547.935714:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564036096:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826547.935745:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826547.935755:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826547/real 1787826547] req@ffff976882aee580 x1874670564036096/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826547.935766:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564036096/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826547.935779:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826547.935782:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826547.935786:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564036096:192.168.203.139@tcp:8:kworker.0 00010000:00010000:3.0:1787826548.190943:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826548.959383:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826549.215003:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826549.982957:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826550.238960:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826551.007379:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826551.263053:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826552.031288:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826552.287381:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787826553.054987:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826553.054997:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787826553.055002:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826553.055013:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1492 00000100:00080000:2.0:1787826553.055024:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787826553.055031:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787826553.055032:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826553.055071:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9a40 x1874670564036224/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826553.055098:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826553.055102:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826553.055104:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826553.055107:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1492 00000100:00080000:2.0:1787826553.055111:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826553.055114:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826553.055124:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826553.055139:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564036352/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826553.055144:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564036224:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787826553.055150:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826553.055154:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787826553.055155:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826553.055158:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1492 00000100:00080000:2.0:1787826553.055160:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826553.055162:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826553.055166:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826553.055174:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564036480/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826553.055180:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826553.055182:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826553.055183:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826553.055186:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1492 00000100:00080000:2.0:1787826553.055188:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826553.055190:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826553.055192:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787826553.055198:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564036608/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826553.055204:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826553.055205:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826553.055207:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787826553.055213:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787826553.055216:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1492 00000100:00080000:2.0:1787826553.055220:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826553.055224:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826553.055230:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787826553.055241:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564036736/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826553.055258:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826553/real 1787826553] req@ffff976889cf9a40 x1874670564036224/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826553.055283:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9a40 x1874670564036224/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826553.055294:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826553.055312:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826553.055316:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564036224:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826553.055339:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564036352:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826553.055362:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826553.055374:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826553/real 1787826553] req@ffff976889cfb0c0 x1874670564036352/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826553.055385:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564036352/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826553.055392:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826553.055395:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826553.055398:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564036352:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826553.055403:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564036480:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826553.055419:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826553.055428:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826553/real 1787826553] req@ffff976889cf8000 x1874670564036480/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826553.055438:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564036480/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826553.055449:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826553.055452:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826553.055455:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564036480:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826553.055460:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564036608:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826553.055476:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826553.055485:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826553/real 1787826553] req@ffff976889cfbc00 x1874670564036608/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826553.055495:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564036608/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826553.055502:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826553.055506:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826553.055509:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564036608:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826553.055515:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564036736:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826553.055532:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826553.055543:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826553/real 1787826553] req@ffff976889cfa940 x1874670564036736/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826553.055553:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564036736/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826553.055560:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826553.055563:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826553.055566:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564036736:192.168.203.139@tcp:8:kworker.0 00010000:00010000:2.0:1787826553.311032:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826554.078995:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826554.334950:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826555.102921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826555.359023:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826556.126894:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826556.383133:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826557.150943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826557.407003:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826558.174982:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787826558.174996:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826558.175003:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826558.175007:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826558.175016:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1498 00000100:00080000:0.0:1787826558.175026:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826558.175033:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787826558.175073:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564036864/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826558.175098:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826558.175102:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826558.175103:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826558.175107:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1498 00000100:00080000:0.0:1787826558.175111:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826558.175115:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826558.175125:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826558.175134:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564036864:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826558.175140:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564036992/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826558.175152:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826558.175160:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826558.175162:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826558.175165:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1498 00000100:00080000:0.0:1787826558.175169:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826558.175182:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826558.175190:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787826558.175195:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826558.175205:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564037120/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826558.175214:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826558.175217:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826558.175219:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826558.175223:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1498 00000100:00080000:0.0:1787826558.175227:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826558.175231:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826558.175236:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787826558.175238:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826558/real 1787826558] req@ffff976882aee940 x1874670564036864/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826558.175249:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564037248/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826558.175258:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564036864/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826558.175274:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826558.175283:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826558.175284:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826558.175285:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826558.175290:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:0.0:1787826558.175293:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1498 00000100:00080000:0.0:1787826558.175298:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826558.175302:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826558.175306:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564036864:192.168.203.139@tcp:250:kworker.0 00000008:00100000:0.0:1787826558.175308:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826558.175322:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564037376/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826558.175341:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564036992:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826558.175373:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826558.175385:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826558/real 1787826558] req@ffff976882aef0c0 x1874670564036992/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826558.175402:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564036992/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826558.175409:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826558.175412:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826558.175415:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564036992:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826558.175420:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564037120:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826558.175439:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826558.175449:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826558/real 1787826558] req@ffff976882aee580 x1874670564037120/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826558.175460:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564037120/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826558.175469:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826558.175471:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826558.175474:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564037120:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826558.175487:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564037248:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826558.175502:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826558.175509:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826558/real 1787826558] req@ffff976882aeed00 x1874670564037248/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826558.175515:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564037248/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826558.175524:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826558.175526:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826558.175528:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564037248:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826558.175538:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564037376:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826558.175548:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826558.175557:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826558/real 1787826558] req@ffff976882aec3c0 x1874670564037376/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826558.175567:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564037376/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826558.175575:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826558.175577:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826558.175580:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564037376:192.168.203.139@tcp:8:kworker.0 00010000:00010000:2.0:1787826558.431074:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826559.198931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826559.454969:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826560.223004:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826560.478975:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826561.246921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826561.503051:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826562.270957:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826562.526983:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826563.294952:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826563.294965:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826563.294969:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826563.294979:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1503 00000100:00080000:0.0:1787826563.294990:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787826563.294993:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787826563.294999:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787826563.295036:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564037504/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826563.295059:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826563.295062:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826563.295064:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826563.295074:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1503 00000100:00080000:0.0:1787826563.295078:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826563.295084:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826563.295094:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826563.295103:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564037504:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826563.295105:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef840 x1874670564037632/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826563.295113:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826563.295116:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826563.295117:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826563.295120:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1503 00000100:00080000:0.0:1787826563.295123:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826563.295126:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826563.295130:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826563.295139:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564037760/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826563.295156:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787826563.295158:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787826563.295159:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826563.295160:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826563.295164:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1503 00000100:00080000:0.0:1787826563.295167:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826563.295169:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826563.295174:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787826563.295182:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564037888/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826563.295190:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787826563.295191:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826563/real 1787826563] req@ffff976882aed680 x1874670564037504/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826563.295193:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826563.295195:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826563.295198:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1503 00000100:00080000:0.0:1787826563.295200:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826563.295203:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826563.295206:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564037504/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787826563.295210:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826563.295218:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564038016/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826563.295221:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826563.295237:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826563.295241:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564037504:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826563.295261:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564037632:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826563.295281:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826563.295290:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826563/real 1787826563] req@ffff976882aef840 x1874670564037632/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826563.295297:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef840 x1874670564037632/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826563.295302:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826563.295304:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826563.295306:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564037632:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826563.295310:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564037760:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826563.295320:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826563.295325:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826563/real 1787826563] req@ffff976882aec3c0 x1874670564037760/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826563.295331:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564037760/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826563.295336:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826563.295338:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826563.295339:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564037760:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826563.295343:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564037888:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826563.295352:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826563.295360:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826563/real 1787826563] req@ffff976882aeed00 x1874670564037888/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826563.295372:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564037888/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826563.295378:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826563.295380:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826563.295383:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564037888:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826563.295388:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564038016:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826563.295404:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826563.295413:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826563/real 1787826563] req@ffff976882aee580 x1874670564038016/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826563.295422:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564038016/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826563.295429:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826563.295431:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826563.295434:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564038016:192.168.203.139@tcp:8:kworker.0 00010000:00010000:3.0:1787826563.550981:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826564.318976:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826564.575089:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826565.343049:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826565.599500:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826566.367123:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826566.623183:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826567.390939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826567.647003:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826568.414977:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787826568.414982:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826568.414993:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826568.414999:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826568.415011:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1508 00000100:00080000:0.0:1787826568.415026:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826568.415035:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787826568.415083:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564038144/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826568.415122:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826568.415126:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826568.415129:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826568.415133:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1508 00000100:00080000:0.0:1787826568.415137:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826568.415145:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826568.415158:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826568.415172:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564038272/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826568.415183:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787826568.415184:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564038144:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787826568.415204:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826568.415207:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826568.415212:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1508 00000100:00080000:0.0:1787826568.415216:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826568.415220:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826568.415225:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826568.415239:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564038400/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826568.415248:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826568.415252:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826568.415253:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826568.415257:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1508 00000100:00080000:0.0:1787826568.415260:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826568.415264:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826568.415269:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787826568.415280:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564038528/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826568.415291:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787826568.415291:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787826568.415297:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826568.415299:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826568.415303:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1508 00000100:00080000:0.0:1787826568.415308:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826568.415317:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826568.415322:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826568.415334:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564038656/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826568.415351:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826568/real 1787826568] req@ffff976882aef0c0 x1874670564038144/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826568.415373:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564038144/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826568.415385:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826568.415389:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826568.415393:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564038144:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826568.415419:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564038272:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826568.415441:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826568.415455:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826568/real 1787826568] req@ffff976882aee940 x1874670564038272/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826568.415467:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564038272/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826568.415475:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826568.415479:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826568.415482:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564038272:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826568.415488:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564038400:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826568.415509:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826568.415520:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826568/real 1787826568] req@ffff976882aee580 x1874670564038400/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826568.415538:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564038400/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826568.415544:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826568.415547:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826568.415549:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564038400:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826568.415553:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564038528:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826568.415568:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826568.415575:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826568/real 1787826568] req@ffff976882aeed00 x1874670564038528/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826568.415587:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564038528/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826568.415594:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826568.415597:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826568.415600:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564038528:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826568.415605:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564038656:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826568.415624:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826568.415635:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826568/real 1787826568] req@ffff976882aec3c0 x1874670564038656/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826568.415643:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564038656/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826568.415652:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826568.415655:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826568.415657:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564038656:192.168.203.139@tcp:8:kworker.0 00010000:00010000:2.0:1787826568.671012:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826569.438974:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826569.694998:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826570.462913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826570.719003:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826571.486951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826571.743020:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826572.511243:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826572.766953:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787826573.535346:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826573.535353:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826573.535358:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826573.535366:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1513 00000100:00080000:1.0:1787826573.535377:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826573.535384:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826573.535416:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564038784/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826573.535440:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826573.535445:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826573.535447:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826573.535451:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1513 00000100:00080000:1.0:1787826573.535456:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826573.535461:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826573.535470:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564038784:192.168.203.139@tcp:250:kworker.0 00000008:00100000:1.0:1787826573.535472:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826573.535488:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564038912/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787826573.535532:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826573.535877:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826573.536615:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826573.536619:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826573.536622:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826573.536629:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1513 00000100:00080000:1.0:1787826573.536634:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787826573.536635:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826573/real 1787826573] req@ffff976889efe940 x1874670564038784/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826573.536642:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826573.536654:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826573.536663:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564038784/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826573.536672:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564039040/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826573.536676:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826573.536686:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:1.0:1787826573.536692:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787826573.536693:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564038784:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787826573.536697:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826573.536702:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826573.536707:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1513 00000100:00080000:1.0:1787826573.536711:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826573.536714:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826573.536719:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787826573.536723:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564038912:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826573.536732:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564039168/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826573.536743:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826573.536746:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826573.536748:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826573.536751:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1513 00000100:00080000:1.0:1787826573.536754:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826573.536760:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:2.0:1787826573.536760:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787826573.536767:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787826573.536782:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826573/real 1787826573] req@ffff976889eff480 x1874670564038912/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826573.536783:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd2c0 x1874670564039296/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826573.536812:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564038912/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826573.536888:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826573.536892:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826573.536896:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564038912:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826573.536903:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564039040:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826573.536924:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826573.536934:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826573/real 1787826573] req@ffff976889efe580 x1874670564039040/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826573.536945:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564039040/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826573.536953:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826573.536957:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826573.536960:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564039040:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826573.536978:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564039168:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826573.536999:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826573.537009:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826573/real 1787826573] req@ffff976889efcb40 x1874670564039168/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826573.537020:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564039168/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826573.537028:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826573.537031:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826573.537034:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564039168:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826573.537040:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564039296:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826573.537053:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826573.537061:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826573/real 1787826573] req@ffff976889efd2c0 x1874670564039296/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826573.537071:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd2c0 x1874670564039296/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826573.537079:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826573.537081:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826573.537084:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564039296:192.168.203.139@tcp:8:kworker.0 00010000:00010000:2.0:1787826573.791203:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826574.558941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826574.815106:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826575.582936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826575.839050:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826576.607181:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826576.863015:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826577.630985:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826577.887454:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826578.654912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787826578.654931:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826578.654937:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787826578.654947:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826578.654957:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1518 00000100:00080000:2.0:1787826578.654967:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826578.654974:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826578.655007:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564039424/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826578.655030:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826578.655034:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826578.655035:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826578.655038:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1518 00000100:00080000:2.0:1787826578.655041:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826578.655044:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826578.655052:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826578.655061:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564039424:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826578.655062:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564039552/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826578.655079:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826578.655082:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787826578.655084:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826578.655088:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1518 00000100:00080000:2.0:1787826578.655091:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826578.655095:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826578.655100:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826578.655113:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564039680/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826578.655134:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826578.655137:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826578.655139:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826578.655143:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1518 00000100:00080000:2.0:1787826578.655147:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826578.655151:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826578.655158:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787826578.655164:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564039552:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826578.655172:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564039808/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826578.655181:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826578.655183:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826578.655185:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826578.655188:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1518 00000100:00100000:0.0:1787826578.655191:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564039680:192.168.203.139@tcp:38:kworker.0 00000100:00080000:2.0:1787826578.655191:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826578.655197:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826578.655201:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787826578.655209:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564039936/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826578.655221:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564039808:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826578.655236:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564039936:192.168.203.139@tcp:8:kworker.0 00010000:00010000:3.0:1787826578.911164:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826579.678987:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826579.936098:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826580.703206:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826580.959033:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787826581.407246:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:3.0:1787826581.407300:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787826581.407319:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826581.407360:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826581.407370:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826581.407378:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826581.407385:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:0.0:1787826581.407417:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826578/real 1787826581] req@ffff976889cfbc00 x1874670564039808/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826581.407445:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564039808/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826581.407459:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826581.407465:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826581.407470:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564039808:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826581.407483:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826578/real 1787826581] req@ffff976889cf8000 x1874670564039936/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826581.407500:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564039936/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826581.407508:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826581.407511:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826581.407514:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564039936:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826581.407520:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826578/real 1787826581] req@ffff976889cf8f00 x1874670564039552/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826581.407534:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564039552/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826581.407541:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826581.407544:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826581.407547:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564039552:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826581.407553:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826578/real 1787826581] req@ffff976889cfa940 x1874670564039680/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826581.407564:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564039680/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826581.407571:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826581.407574:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826581.407577:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564039680:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826581.407583:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826578/real 1787826581] req@ffff976889cf9e00 x1874670564039424/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826581.407594:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564039424/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826581.407601:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826581.407604:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826581.407607:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564039424:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826581.726980:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826581.982983:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826582.750942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826583.006979:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826583.775121:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826583.775166:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826583.775182:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826583.775187:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826583.775199:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1523 00000100:00080000:1.0:1787826583.775210:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826583.775218:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826583.775255:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564040064/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826583.775290:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826583.775293:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826583.775295:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826583.775300:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1523 00000100:00080000:1.0:1787826583.775304:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826583.775308:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826583.775319:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826583.775333:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564040192/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826583.775336:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564040064:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787826583.775352:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826583.775356:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826583.775358:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826583.775372:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1523 00000100:00080000:1.0:1787826583.775394:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826583.775398:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:0.0:1787826583.775404:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787826583.775405:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826583.775420:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efda40 x1874670564040320/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826583.775429:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826583.775433:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826583.775435:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826583.775439:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1523 00000100:00080000:1.0:1787826583.775443:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787826583.775445:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826583/real 1787826583] req@ffff976889efc780 x1874670564040064/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826583.775449:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826583.775456:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787826583.775473:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564040064/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826583.775473:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd2c0 x1874670564040448/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826583.775498:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826583.775502:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826583.775504:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826583.775505:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826583.775511:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:1.0:1787826583.775513:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1523 00000100:00080000:1.0:1787826583.775519:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787826583.775520:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564040064:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787826583.775525:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826583.775532:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787826583.775544:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564040576/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826583.775550:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564040192:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826583.775577:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826583.775590:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826583/real 1787826583] req@ffff976889efe1c0 x1874670564040192/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826583.775602:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564040192/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826583.775610:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826583.775613:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826583.775616:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564040192:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826583.775622:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564040320:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826583.775641:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826583.775652:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826583/real 1787826583] req@ffff976889efda40 x1874670564040320/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826583.775670:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efda40 x1874670564040320/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826583.775677:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826583.775680:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826583.775683:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564040320:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826583.775688:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564040448:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826583.775706:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826583.775716:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826583/real 1787826583] req@ffff976889efd2c0 x1874670564040448/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826583.775727:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd2c0 x1874670564040448/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826583.775734:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826583.775736:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826583.775739:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564040448:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826583.775756:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564040576:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826583.775773:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826583.775783:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826583/real 1787826583] req@ffff976889efcb40 x1874670564040576/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826583.775793:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564040576/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826583.775805:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826583.775808:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826583.775810:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564040576:192.168.203.139@tcp:8:kworker.0 00010000:00010000:2.0:1787826584.031012:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826584.798986:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826585.054950:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826585.822939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826586.079179:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826586.846952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826587.103030:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826587.870984:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826588.126991:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826588.894950:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787826588.894950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787826588.894960:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826588.894965:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826588.894973:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1528 00000100:00080000:0.0:1787826588.894984:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826588.894991:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787826588.895022:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef840 x1874670564040704/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826588.895057:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826588.895061:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826588.895063:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826588.895068:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1528 00000100:00080000:0.0:1787826588.895072:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826588.895076:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826588.895086:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826588.895089:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564040704:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826588.895099:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564040832/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826588.895110:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826588.895113:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826588.895116:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826588.895120:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1528 00000100:00080000:0.0:1787826588.895123:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826588.895127:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826588.895132:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826588.895143:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564040960/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826588.895155:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826588.895158:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826588.895161:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787826588.895164:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787826588.895168:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1528 00000100:00080000:0.0:1787826588.895173:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826588.895177:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826588.895185:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787826588.895197:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564041088/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826588.895206:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787826588.895210:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826588/real 1787826588] req@ffff976882aef840 x1874670564040704/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826588.895211:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826588.895214:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826588.895223:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1528 00000100:00080000:0.0:1787826588.895229:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826588.895233:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826588.895237:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787826588.895238:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef840 x1874670564040704/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826588.895251:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564041216/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826588.895269:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826588.895275:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826588.895281:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564040704:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826588.895303:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564040832:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826588.895333:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826588.895346:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826588/real 1787826588] req@ffff976882aed680 x1874670564040832/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826588.895357:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564040832/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826588.895364:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826588.895367:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826588.895370:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564040832:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826588.895376:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564040960:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826588.895400:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826588.895410:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826588/real 1787826588] req@ffff976882aec3c0 x1874670564040960/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826588.895426:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564040960/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826588.895434:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826588.895436:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826588.895439:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564040960:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826588.895445:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564041088:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826588.895472:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826588.895484:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826588/real 1787826588] req@ffff976882aeed00 x1874670564041088/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826588.895495:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564041088/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826588.895501:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826588.895504:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826588.895507:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564041088:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826588.895512:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564041216:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826588.895538:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826588.895548:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826588/real 1787826588] req@ffff976882aee580 x1874670564041216/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826588.895555:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564041216/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826588.895563:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826588.895565:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826588.895567:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564041216:192.168.203.139@tcp:8:kworker.0 00010000:00010000:2.0:1787826589.150993:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826589.919035:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826590.175065:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826590.942986:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826591.199047:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826591.966939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826592.223370:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826592.990965:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826593.247066:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826594.014915:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787826594.014918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787826594.014923:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826594.014927:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826594.014934:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1533 00000100:00080000:0.0:1787826594.014943:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826594.014949:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787826594.014981:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564041344/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826594.015002:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826594.015005:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826594.015007:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826594.015021:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1533 00000100:00080000:0.0:1787826594.015025:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826594.015030:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826594.015040:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826594.015048:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564041344:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826594.015054:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564041472/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826594.015064:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826594.015067:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826594.015069:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826594.015073:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1533 00000100:00080000:0.0:1787826594.015076:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826594.015080:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826594.015084:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826594.015094:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564041600/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826594.015100:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826594.015102:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826594.015103:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826594.015106:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1533 00000100:00080000:0.0:1787826594.015108:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826594.015110:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826594.015113:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787826594.015120:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564041728/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787826594.015120:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787826594.015128:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826594.015130:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826594.015131:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826594.015134:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1533 00000100:00080000:0.0:1787826594.015136:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826594.015138:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826594.015141:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826594.015146:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564041856/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826594.015166:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826594/real 1787826594] req@ffff976882aee940 x1874670564041344/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826594.015184:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564041344/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826594.015198:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826594.015202:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826594.015206:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564041344:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826594.015227:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564041472:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826594.015249:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826594.015262:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826594/real 1787826594] req@ffff976882aef0c0 x1874670564041472/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826594.015280:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564041472/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826594.015288:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826594.015291:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826594.015294:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564041472:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826594.015301:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564041600:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826594.015319:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826594.015329:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826594/real 1787826594] req@ffff976882aee580 x1874670564041600/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826594.015340:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564041600/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826594.015347:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826594.015350:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826594.015353:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564041600:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826594.015358:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564041728:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826594.015377:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826594.015387:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826594/real 1787826594] req@ffff976882aeed00 x1874670564041728/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826594.015399:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564041728/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826594.015412:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826594.015415:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826594.015418:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564041728:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826594.015423:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564041856:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826594.015445:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826594.015456:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826594/real 1787826594] req@ffff976882aec3c0 x1874670564041856/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826594.015466:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564041856/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826594.015473:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826594.015476:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826594.015479:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564041856:192.168.203.139@tcp:8:kworker.0 00010000:00010000:3.0:1787826594.270977:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826595.038944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826595.295035:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826596.062955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826596.319065:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826597.086947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826597.342967:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826598.110914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826598.367001:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787826599.134950:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787826599.134952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787826599.134959:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787826599.134964:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826599.134983:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1539 00000100:00080000:2.0:1787826599.134993:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826599.135002:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826599.135038:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564041984/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826599.135062:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826599.135064:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826599.135065:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826599.135068:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1539 00000100:00080000:2.0:1787826599.135070:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826599.135073:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826599.135084:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826599.135091:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9a40 x1874670564042112/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826599.135097:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826599.135100:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787826599.135103:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787826599.135104:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564041984:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787826599.135108:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1539 00000100:00080000:2.0:1787826599.135113:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826599.135115:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826599.135118:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826599.135125:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564042240/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826599.135130:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826599.135132:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826599.135133:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826599.135135:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1539 00000100:00080000:2.0:1787826599.135138:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826599.135140:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826599.135144:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787826599.135150:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564042368/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826599.135155:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826599.135157:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826599.135159:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826599.135161:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1539 00000100:00080000:2.0:1787826599.135163:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826599.135165:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826599.135167:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787826599.135174:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564042496/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787826599.135181:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826599.135228:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826599/real 1787826599] req@ffff976889cfb0c0 x1874670564041984/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826599.135249:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564041984/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826599.135258:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826599.135261:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826599.135265:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564041984:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826599.135286:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564042112:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826599.135307:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826599.135318:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826599/real 1787826599] req@ffff976889cf9a40 x1874670564042112/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826599.135328:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9a40 x1874670564042112/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826599.135335:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826599.135338:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826599.135341:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564042112:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826599.135346:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564042240:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826599.135363:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826599.135373:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826599/real 1787826599] req@ffff976889cfb480 x1874670564042240/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826599.135392:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564042240/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826599.135400:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826599.135402:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826599.135405:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564042240:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826599.135411:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564042368:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826599.135429:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826599.135438:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826599/real 1787826599] req@ffff976889cfa580 x1874670564042368/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826599.135449:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564042368/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826599.135455:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826599.135458:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826599.135461:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564042368:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826599.135466:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564042496:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826599.135485:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826599.135496:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826599/real 1787826599] req@ffff976889cfa1c0 x1874670564042496/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826599.135514:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564042496/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826599.135521:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826599.135524:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826599.135526:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564042496:192.168.203.139@tcp:8:kworker.0 00010000:00010000:1.0:1787826599.391122:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826600.158940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826600.414964:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826601.183163:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826601.439115:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826602.206929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826602.463106:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826603.230928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826603.486993:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826604.254990:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787826604.254997:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826604.255008:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826604.255014:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826604.255025:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1544 00000100:00080000:3.0:1787826604.255038:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826604.255047:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787826604.255097:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564042624/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826604.255126:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826604.255130:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826604.255135:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826604.255139:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1544 00000100:00080000:3.0:1787826604.255150:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826604.255156:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826604.255167:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826604.255179:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564042624:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787826604.255186:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564042752/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826604.255206:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826604.255210:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826604.255212:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826604.255218:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1544 00000100:00080000:3.0:1787826604.255223:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826604.255226:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826604.255232:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826604.255246:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564042880/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826604.255256:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826604.255260:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826604.255262:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826604.255266:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1544 00000100:00080000:3.0:1787826604.255270:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826604.255273:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826604.255278:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1787826604.255285:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826604.255294:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564043008/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826604.255314:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826604.255317:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826604.255320:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826604.255324:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1544 00000100:00080000:3.0:1787826604.255329:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826604.255333:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826604.255340:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826604.255355:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564043136/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826604.255360:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826604/real 1787826604] req@ffff97688cb5de00 x1874670564042624/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826604.255406:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564042624/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826604.255420:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826604.255425:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826604.255429:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564042624:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826604.255461:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564042752:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826604.255487:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826604.255502:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826604/real 1787826604] req@ffff97688cb5f0c0 x1874670564042752/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826604.255516:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564042752/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826604.255524:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826604.255527:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826604.255530:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564042752:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826604.255537:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564042880:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826604.255566:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826604.255578:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826604/real 1787826604] req@ffff97688cb5d680 x1874670564042880/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826604.255590:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564042880/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826604.255598:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826604.255602:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826604.255605:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564042880:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826604.255611:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564043008:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826604.255631:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826604.255642:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826604/real 1787826604] req@ffff97688cb5c000 x1874670564043008/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826604.255653:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564043008/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826604.255665:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826604.255668:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826604.255671:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564043008:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826604.255677:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564043136:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826604.255695:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826604.255706:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826604/real 1787826604] req@ffff97688cb5cb40 x1874670564043136/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826604.255718:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564043136/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826604.255725:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826604.255729:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826604.255732:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564043136:192.168.203.139@tcp:8:kworker.0 00010000:00010000:2.0:1787826604.511006:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826605.278947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826605.534996:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826606.302924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826606.558955:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826607.326934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826607.583092:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826608.350981:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826608.607004:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826609.374947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787826609.374947:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826609.374957:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826609.374961:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826609.374968:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1549 00000100:00080000:3.0:1787826609.374977:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826609.374983:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787826609.375013:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564043264/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826609.375033:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826609.375036:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826609.375038:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826609.375041:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1549 00000100:00080000:3.0:1787826609.375044:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826609.375049:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826609.375058:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826609.375071:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564043392/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826609.375080:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826609.375084:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826609.375085:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826609.375089:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1549 00000100:00080000:3.0:1787826609.375093:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826609.375097:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826609.375102:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826609.375115:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564043520/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826609.375124:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826609.375126:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826609.375128:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826609.375131:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1549 00000100:00080000:3.0:1787826609.375135:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826609.375138:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826609.375142:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787826609.375152:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564043648/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826609.375159:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826609.375163:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826609.375165:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826609.375175:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1549 00000100:00080000:3.0:1787826609.375179:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826609.375184:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826609.375188:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826609.375202:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564043776/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826609.375290:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564043264:192.168.203.139@tcp:250:kworker.0 00000800:00000100:0.0:1787826609.375351:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826609.375387:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826609/real 1787826609] req@ffff97688cb5cf00 x1874670564043264/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826609.375405:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564043264/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826609.375413:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826609.375417:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826609.375421:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564043264:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826609.375429:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564043392:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826609.375451:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826609.375465:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826609/real 1787826609] req@ffff97688cb5c3c0 x1874670564043392/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826609.375478:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564043392/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826609.375486:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826609.375489:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826609.375492:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564043392:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826609.375497:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564043520:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826609.375514:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826609.375522:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826609/real 1787826609] req@ffff97688cb5f840 x1874670564043520/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826609.375530:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564043520/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826609.375535:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826609.375538:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826609.375540:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564043520:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826609.375544:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564043648:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826609.375557:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826609.375567:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826609/real 1787826609] req@ffff97688cb5c780 x1874670564043648/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826609.375578:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564043648/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826609.375586:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826609.375589:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826609.375592:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564043648:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826609.375598:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564043776:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826609.375615:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826609.375622:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826609/real 1787826609] req@ffff97688cb5e1c0 x1874670564043776/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826609.375629:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564043776/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826609.375634:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826609.375636:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826609.375638:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564043776:192.168.203.139@tcp:8:kworker.0 00010000:00010000:3.0:1787826609.631039:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826610.398975:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826610.655042:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826611.422954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826611.679039:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826612.446945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826612.704029:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826613.471002:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826613.727005:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787826614.495152:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787826614.495159:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826614.495161:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826614.495167:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826614.495177:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1554 00000100:00080000:1.0:1787826614.495193:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826614.495200:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826614.495234:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564043904/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826614.495256:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826614.495260:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826614.495261:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826614.495265:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1554 00000100:00080000:1.0:1787826614.495268:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826614.495273:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826614.495283:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826614.495292:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564043904:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826614.495296:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564044032/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826614.495308:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826614.495312:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826614.495314:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826614.495318:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1554 00000100:00080000:1.0:1787826614.495322:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826614.495326:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826614.495331:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826614.495341:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564044160/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826614.495351:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787826614.495355:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787826614.495356:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826614.495360:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826614.495366:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1554 00000100:00080000:1.0:1787826614.495370:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826614.495374:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826614.495379:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826614.495390:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd680 x1874670564044288/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826614.495397:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826614/real 1787826614] req@ffff976889efe580 x1874670564043904/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826614.495404:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826614.495413:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826614.495416:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826614.495421:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1554 00000100:00100000:0.0:1787826614.495426:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564043904/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826614.495426:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826614.495433:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826614.495441:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787826614.495446:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826614.495454:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826614.495456:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564044416/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826614.495461:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564043904:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826614.495490:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564044032:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826614.495514:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826614.495539:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826614/real 1787826614] req@ffff976889eff480 x1874670564044032/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826614.495552:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564044032/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826614.495560:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826614.495563:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826614.495566:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564044032:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826614.495572:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564044160:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826614.495592:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826614.495602:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826614/real 1787826614] req@ffff976889efe940 x1874670564044160/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826614.495613:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564044160/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826614.495621:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826614.495624:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826614.495628:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564044160:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826614.495633:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564044288:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826614.495651:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826614.495660:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826614/real 1787826614] req@ffff976889efd680 x1874670564044288/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826614.495672:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd680 x1874670564044288/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826614.495679:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826614.495682:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826614.495685:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564044288:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826614.495690:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564044416:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826614.495707:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826614.495716:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826614/real 1787826614] req@ffff976889eff0c0 x1874670564044416/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826614.495736:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564044416/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826614.495750:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826614.495753:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826614.495756:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564044416:192.168.203.139@tcp:8:kworker.0 00010000:00010000:1.0:1787826614.750994:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826615.518953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826615.774989:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826616.542958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826616.799015:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826617.566900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826617.822983:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826618.590957:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826618.846965:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826619.614947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787826619.614976:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826619.614982:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787826619.614986:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826619.614996:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1559 00000100:00080000:2.0:1787826619.615005:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826619.615012:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826619.615047:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564044544/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826619.615071:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826619.615074:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826619.615076:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826619.615080:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1559 00000100:00080000:2.0:1787826619.615084:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826619.615088:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826619.615097:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826619.615107:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564044544:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826619.615111:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564044672/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826619.615123:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826619.615126:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787826619.615128:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826619.615131:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1559 00000100:00080000:2.0:1787826619.615134:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826619.615138:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826619.615143:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826619.615154:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564044800/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826619.615163:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826619.615166:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826619.615168:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787826619.615176:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787826619.615181:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1559 00000100:00080000:2.0:1787826619.615187:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826619.615192:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826619.615197:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787826619.615208:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564044928/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826619.615217:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826619.615220:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826619.615222:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826619.615233:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1559 00000100:00100000:0.0:1787826619.615238:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826619/real 1787826619] req@ffff976889cf92c0 x1874670564044544/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826619.615239:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826619.615246:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826619.615255:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826619.615268:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564044544/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826619.615272:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564045056/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826619.615293:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826619.615300:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826619.615304:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564044544:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826619.615328:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564044672:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826619.615352:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826619.615374:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826619/real 1787826619] req@ffff976889cfad00 x1874670564044672/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826619.615387:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564044672/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826619.615396:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826619.615399:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826619.615403:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564044672:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826619.615409:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564044800:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826619.615430:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826619.615441:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826619/real 1787826619] req@ffff976889cfa1c0 x1874670564044800/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826619.615453:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564044800/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826619.615461:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826619.615464:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826619.615467:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564044800:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826619.615472:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564044928:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826619.615491:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826619.615501:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826619/real 1787826619] req@ffff976889cfa580 x1874670564044928/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826619.615513:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564044928/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826619.615526:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826619.615530:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826619.615534:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564044928:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826619.615540:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564045056:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826619.615558:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826619.615568:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826619/real 1787826619] req@ffff976889cfb480 x1874670564045056/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826619.615579:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564045056/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826619.615586:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826619.615589:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826619.615591:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564045056:192.168.203.139@tcp:8:kworker.0 00010000:00010000:2.0:1787826619.871005:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826620.638946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826620.894979:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826621.662951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826621.919028:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826622.686964:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826622.943013:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826623.710934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826623.967004:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826624.734940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787826624.734948:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826624.734954:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826624.734958:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826624.734968:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1564 00000100:00080000:3.0:1787826624.734987:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826624.734995:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787826624.735027:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564045184/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826624.735046:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826624.735048:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826624.735050:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826624.735053:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1564 00000100:00080000:3.0:1787826624.735057:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826624.735060:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826624.735067:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826624.735076:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564045312/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826624.735082:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826624.735085:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826624.735087:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826624.735090:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1564 00000100:00080000:3.0:1787826624.735093:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826624.735097:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826624.735101:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826624.735125:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564045440/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826624.735135:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826624.735137:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826624.735139:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826624.735144:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1564 00000100:00080000:3.0:1787826624.735151:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826624.735155:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826624.735160:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787826624.735177:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564045184:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787826624.735181:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564045568/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826624.735197:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826624.735200:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826624.735202:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826624.735206:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1564 00000100:00080000:3.0:1787826624.735210:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826624.735213:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826624.735218:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826624.735234:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564045696/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787826624.735262:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826624.735587:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826624/real 1787826624] req@ffff97688cb5d2c0 x1874670564045184/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826624.735604:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564045184/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826624.735614:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826624.735617:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826624.735621:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564045184:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826624.735630:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564045312:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826624.735653:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826624.735677:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826624/real 1787826624] req@ffff97688cb5e1c0 x1874670564045312/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826624.735690:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564045312/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826624.735698:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826624.735701:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826624.735704:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564045312:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826624.735710:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564045440:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826624.735729:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826624.735739:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826624/real 1787826624] req@ffff97688cb5c780 x1874670564045440/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826624.735751:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564045440/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826624.735759:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826624.735762:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826624.735765:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564045440:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826624.735793:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564045568:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826624.735865:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826624.735879:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826624/real 1787826624] req@ffff97688cb5f840 x1874670564045568/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826624.735891:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564045568/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826624.735900:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826624.735903:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826624.735906:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564045568:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826624.735911:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564045696:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826624.735939:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826624.735950:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826624/real 1787826624] req@ffff97688cb5c3c0 x1874670564045696/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826624.735962:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564045696/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826624.735969:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826624.735973:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826624.735976:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564045696:192.168.203.139@tcp:8:kworker.0 00010000:00010000:3.0:1787826624.991040:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826625.759013:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826626.015070:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826626.782975:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826627.039052:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826627.806926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826628.062925:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826628.830948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826629.087003:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826629.854967:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787826629.854979:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787826629.854985:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826629.854991:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826629.855001:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1569 00000100:00080000:0.0:1787826629.855013:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826629.855021:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787826629.855057:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564045824/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826629.855085:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826629.855088:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826629.855090:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826629.855094:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1569 00000100:00080000:0.0:1787826629.855097:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826629.855100:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826629.855109:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826629.855127:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef840 x1874670564045952/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826629.855135:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826629.855138:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826629.855140:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826629.855144:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1569 00000100:00080000:0.0:1787826629.855147:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826629.855150:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826629.855154:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826629.855178:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564046080/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826629.855187:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826629.855190:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826629.855192:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826629.855196:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1569 00000100:00080000:0.0:1787826629.855209:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826629.855214:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826629.855219:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787826629.855220:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564045824:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826629.855234:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564046208/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826629.855246:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826629.855249:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826629.855251:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826629.855255:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1569 00000100:00080000:0.0:1787826629.855258:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826629.855262:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826629.855266:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826629.855277:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564046336/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787826629.855286:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826629.855483:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826629/real 1787826629] req@ffff976882aed680 x1874670564045824/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826629.855496:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564045824/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826629.855504:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826629.855508:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826629.855510:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564045824:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826629.855516:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564045952:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826629.855534:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826629.855545:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826629/real 1787826629] req@ffff976882aef840 x1874670564045952/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826629.855557:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef840 x1874670564045952/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826629.855565:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826629.855568:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826629.855570:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564045952:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826629.855576:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564046080:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826629.855595:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826629.855606:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826629/real 1787826629] req@ffff976882aec3c0 x1874670564046080/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826629.855623:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564046080/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826629.855630:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826629.855633:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826629.855636:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564046080:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826629.855661:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564046208:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826629.855683:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826629.855694:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826629/real 1787826629] req@ffff976882aeed00 x1874670564046208/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826629.855705:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564046208/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826629.855712:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826629.855715:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826629.855718:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564046208:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826629.855723:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564046336:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826629.855740:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826629.855771:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826629/real 1787826629] req@ffff976882aee580 x1874670564046336/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826629.855783:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564046336/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826629.855800:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826629.855803:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826629.855806:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564046336:192.168.203.139@tcp:8:kworker.0 00010000:00010000:1.0:1787826630.110972:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826630.878906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826631.135098:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826631.902995:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826632.159523:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826632.926922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826633.183034:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826633.950938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826634.207256:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826634.975129:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787826634.975185:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826634.975191:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826634.975195:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826634.975204:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1574 00000100:00080000:3.0:1787826634.975214:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826634.975221:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787826634.975254:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564046464/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826634.975318:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564046464:192.168.203.139@tcp:250:kworker.0 00000800:00000100:2.0:1787826634.975375:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787826634.975453:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826634.975457:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826634.975460:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826634.975464:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1574 00000100:00080000:3.0:1787826634.975468:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826634.975472:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826634.975481:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826634.975494:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564046592/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826634.975556:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826634/real 1787826634] req@ffff97688cb5cf00 x1874670564046464/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826634.975570:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564046464/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826634.975579:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826634.975583:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826634.975586:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564046464:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826634.975608:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564046592:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826634.975630:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826634.975641:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826634/real 1787826634] req@ffff97688cb5cb40 x1874670564046592/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826634.975658:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564046592/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826634.975666:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826634.975669:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826634.975672:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564046592:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787826634.975913:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826634.975917:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826634.975919:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826634.975923:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1574 00000100:00080000:3.0:1787826634.975928:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826634.975932:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826634.975937:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826634.976001:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564046720/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826634.976040:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564046720:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826634.976065:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787826634.976085:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826634.976089:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826634.976091:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826634.976095:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1574 00000100:00080000:3.0:1787826634.976099:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826634.976103:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826634.976109:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787826634.976122:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564046848/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826634.976158:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826634/real 1787826634] req@ffff97688cb5c000 x1874670564046720/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826634.976169:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564046720/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826634.976176:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826634.976180:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826634.976183:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564046720:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826634.976197:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564046848:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826634.976217:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826634.976228:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826634/real 1787826634] req@ffff97688cb5d680 x1874670564046848/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826634.976238:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564046848/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826634.976252:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826634.976255:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826634.976258:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564046848:192.168.203.139@tcp:8:kworker.0 00000100:00080000:3.0:1787826634.976398:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826634.976402:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826634.976405:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826634.976409:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1574 00000100:00080000:3.0:1787826634.976413:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826634.976417:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826634.976423:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826634.976435:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564046976/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826634.976463:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564046976:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826634.976486:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826634.976496:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826634/real 1787826634] req@ffff97688cb5f0c0 x1874670564046976/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826634.976503:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564046976/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826634.976508:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826634.976510:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826634.976512:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564046976:192.168.203.139@tcp:8:kworker.0 00010000:00010000:2.0:1787826635.231016:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826635.998936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826636.255807:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826637.022992:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826637.279055:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826638.047278:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826638.303772:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826639.070947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826639.326974:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826640.094958:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826640.094969:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826640.094974:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826640.094986:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1579 00000100:00080000:0.0:1787826640.094998:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787826640.095003:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787826640.095007:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787826640.095050:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564047104/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826640.095079:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826640.095083:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826640.095085:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826640.095090:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1579 00000100:00080000:0.0:1787826640.095093:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826640.095098:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826640.095108:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826640.095112:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564047104:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826640.095125:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564047232/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826640.095136:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826640.095140:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826640.095142:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826640.095146:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1579 00000100:00080000:0.0:1787826640.095150:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826640.095154:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826640.095160:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826640.095184:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564047360/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826640.095193:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826640.095196:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826640.095198:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826640.095202:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1579 00000100:00080000:0.0:1787826640.095206:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787826640.095210:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787826640.095215:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826640.095222:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787826640.095239:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564047488/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826640.095248:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826640.095251:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826640.095253:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826640.095258:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1579 00000100:00080000:0.0:1787826640.095261:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826640.095265:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826640.095270:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787826640.095274:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826640/real 1787826640] req@ffff976882aef0c0 x1874670564047104/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826640.095287:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564047616/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826640.095302:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564047104/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826640.095316:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826640.095322:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826640.095327:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564047104:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826640.095363:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564047232:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826640.095390:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826640.095416:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826640/real 1787826640] req@ffff976882aee940 x1874670564047232/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826640.095429:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564047232/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826640.095438:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826640.095442:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826640.095445:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564047232:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826640.095452:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564047360:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826640.095472:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826640.095482:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826640/real 1787826640] req@ffff976882aee580 x1874670564047360/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826640.095493:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564047360/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826640.095501:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826640.095505:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826640.095508:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564047360:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826640.095516:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564047488:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826640.095537:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826640.095562:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826640/real 1787826640] req@ffff976882aeed00 x1874670564047488/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826640.095575:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564047488/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826640.095584:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826640.095588:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826640.095591:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564047488:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826640.095597:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564047616:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826640.095618:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826640.095629:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826640/real 1787826640] req@ffff976882aec3c0 x1874670564047616/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826640.095639:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564047616/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826640.095647:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826640.095650:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826640.095654:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564047616:192.168.203.139@tcp:8:kworker.0 00010000:00010000:3.0:1787826640.351069:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826641.118993:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826641.375088:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826642.142925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826642.399123:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826643.167329:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826643.422984:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826644.190947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826644.447003:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826645.214981:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787826645.215368:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826645.215375:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826645.215379:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826645.215388:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1585 00000100:00080000:3.0:1787826645.215397:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826645.215404:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787826645.215444:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564047744/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826645.215472:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826645.215476:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826645.215484:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826645.215489:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1585 00000100:00080000:3.0:1787826645.215492:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826645.215496:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826645.215505:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826645.215516:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564047872/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826645.215525:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826645.215527:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826645.215528:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826645.215531:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1585 00000100:00080000:3.0:1787826645.215534:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826645.215537:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826645.215541:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826645.215550:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564048000/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826645.215558:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826645.215560:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826645.215562:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826645.215565:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1585 00000100:00080000:3.0:1787826645.215567:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826645.215571:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826645.215575:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787826645.215584:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564048128/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826645.215591:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826645.215594:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826645.215596:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826645.215599:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1585 00000100:00080000:3.0:1787826645.215601:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826645.215604:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826645.215608:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826645.215615:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564048256/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826645.215727:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564047744:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826645.215805:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564047872:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826645.215853:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564048000:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826645.215872:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564048128:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826645.215889:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564048256:192.168.203.139@tcp:8:kworker.0 00010000:00010000:1.0:1787826645.471019:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826646.238935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826646.494976:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826647.262949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826647.519023:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787826648.287123:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:3.0:1787826648.287170:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787826648.287187:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826648.287220:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826648.287229:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826648.287236:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826648.287243:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:2.0:1787826648.289213:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826645/real 1787826648] req@ffff97688cb5de00 x1874670564047744/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826648.289243:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564047744/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826648.289255:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826648.289260:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826648.289264:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564047744:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826648.289271:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826645/real 1787826648] req@ffff97688cb5f0c0 x1874670564047872/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826648.289283:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564047872/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826648.289296:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826648.289300:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826648.289303:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564047872:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826648.289310:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826645/real 1787826648] req@ffff97688cb5d680 x1874670564048000/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826648.289322:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564048000/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826648.289329:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826648.289332:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826648.289335:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564048000:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826648.289341:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826645/real 1787826648] req@ffff97688cb5c000 x1874670564048128/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826648.289353:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564048128/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826648.289359:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826648.289363:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826648.289366:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564048128:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826648.289373:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826645/real 1787826648] req@ffff97688cb5cb40 x1874670564048256/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826648.289383:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564048256/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826648.289390:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826648.289394:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826648.289396:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564048256:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826648.290020:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826648.543051:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826649.311738:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826649.566994:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826650.334953:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826650.334964:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1787826650.334964:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787826650.334969:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826650.334979:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1590 00000100:00080000:0.0:1787826650.334989:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826650.334995:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787826650.335023:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef840 x1874670564048384/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826650.335057:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826650.335061:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826650.335063:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826650.335067:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1590 00000100:00080000:0.0:1787826650.335070:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826650.335074:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826650.335083:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826650.335094:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564048512/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826650.335102:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826650.335105:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826650.335107:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826650.335110:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1590 00000100:00080000:0.0:1787826650.335113:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826650.335117:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826650.335122:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826650.335133:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564048640/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826650.335141:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826650.335143:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826650.335145:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826650.335147:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1590 00000100:00080000:0.0:1787826650.335150:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826650.335153:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826650.335156:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787826650.335163:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564048768/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826650.335168:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826650.335171:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826650.335172:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826650.335176:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1590 00000100:00080000:0.0:1787826650.335179:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826650.335182:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826650.335186:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826650.335196:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564048896/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826650.336018:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564048384:192.168.203.139@tcp:250:kworker.0 00000800:00000100:2.0:1787826650.336087:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826650.336144:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826650/real 1787826650] req@ffff976882aef840 x1874670564048384/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826650.336161:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef840 x1874670564048384/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826650.336172:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826650.336176:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826650.336180:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564048384:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826650.336187:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564048512:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826650.336210:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826650.336223:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826650/real 1787826650] req@ffff976882aed680 x1874670564048512/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826650.336234:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564048512/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826650.336241:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826650.336244:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826650.336248:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564048512:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826650.336254:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564048640:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826650.336390:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826650.336401:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826650/real 1787826650] req@ffff976882aec3c0 x1874670564048640/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826650.336413:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564048640/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826650.336421:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826650.336424:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826650.336427:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564048640:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826650.336434:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564048768:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826650.336451:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826650.336460:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826650/real 1787826650] req@ffff976882aeed00 x1874670564048768/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826650.336470:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564048768/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826650.336477:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826650.336480:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826650.336482:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564048768:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826650.336487:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564048896:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826650.336501:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826650.336509:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826650/real 1787826650] req@ffff976882aee580 x1874670564048896/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826650.336519:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564048896/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826650.336526:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826650.336529:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826650.336531:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564048896:192.168.203.139@tcp:8:kworker.0 00010000:00010000:2.0:1787826650.592475:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826651.358951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826651.615761:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826652.383151:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826652.638973:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826653.406939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826653.663327:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826654.431338:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826654.687014:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826655.454930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826655.454982:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826655.454998:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826655.455002:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826655.455010:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1595 00000100:00080000:1.0:1787826655.455020:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826655.455028:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826655.455074:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564049024/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826655.455111:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826655.455114:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826655.455116:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826655.455120:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1595 00000100:00080000:1.0:1787826655.455124:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826655.455128:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826655.455138:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826655.455151:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564049152/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826655.455158:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564049024:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787826655.455173:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826655.455176:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826655.455178:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826655.455183:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1595 00000100:00080000:1.0:1787826655.455187:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826655.455190:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826655.455195:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826655.455207:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564049280/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826655.455216:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826655.455220:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826655.455223:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787826655.455223:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787826655.455231:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1595 00000100:00080000:1.0:1787826655.455236:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826655.455240:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826655.455245:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826655.455259:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564049408/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826655.455266:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826655/real 1787826655] req@ffff976889efc3c0 x1874670564049024/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826655.455294:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826655.455298:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787826655.455300:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564049024/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826655.455301:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826655.455311:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1595 00000100:00080000:1.0:1787826655.455318:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826655.455320:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826655.455324:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:2.0:1787826655.455328:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000008:00100000:1.0:1787826655.455332:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787826655.455340:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564049024:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826655.455347:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd680 x1874670564049536/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826655.455369:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564049152:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826655.455392:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826655.455404:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826655/real 1787826655] req@ffff976889efed00 x1874670564049152/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826655.455422:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564049152/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826655.455430:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826655.455433:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826655.455436:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564049152:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826655.455442:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564049280:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826655.455459:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826655.455469:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826655/real 1787826655] req@ffff976889effc00 x1874670564049280/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826655.455480:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564049280/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826655.455487:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826655.455490:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826655.455494:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564049280:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826655.455499:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564049408:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826655.455516:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826655.455525:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826655/real 1787826655] req@ffff976889eff0c0 x1874670564049408/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826655.455536:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564049408/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826655.455548:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826655.455551:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826655.455554:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564049408:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826655.455559:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564049536:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826655.455576:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826655.455601:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826655/real 1787826655] req@ffff976889efd680 x1874670564049536/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826655.455612:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd680 x1874670564049536/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826655.455620:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826655.455623:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826655.455626:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564049536:192.168.203.139@tcp:8:kworker.0 00010000:00010000:3.0:1787826655.710989:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826656.478949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826656.735108:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826657.503054:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826657.759159:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826658.526936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826658.782962:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826659.550919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826659.806988:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787826660.574951:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787826660.574958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826660.574961:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826660.574973:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826660.574991:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1600 00000100:00080000:1.0:1787826660.575002:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826660.575009:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826660.575043:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564049664/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826660.575070:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826660.575073:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826660.575074:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826660.575078:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1600 00000100:00080000:1.0:1787826660.575081:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826660.575084:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826660.575093:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826660.575105:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564049792/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826660.575115:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826660.575119:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787826660.575120:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564049664:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787826660.575121:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826660.575129:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1600 00000100:00080000:1.0:1787826660.575133:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826660.575136:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826660.575140:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826660.575150:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564049920/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826660.575171:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826660.575175:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826660.575177:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826660.575182:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1600 00000100:00080000:1.0:1787826660.575186:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826660.575189:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826660.575195:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1787826660.575202:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826660.575207:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564050048/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826660.575217:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826660.575221:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826660.575223:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826660.575226:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1600 00000100:00080000:1.0:1787826660.575229:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826660.575232:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826660.575237:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787826660.575246:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd2c0 x1874670564050176/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826660.575262:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826660/real 1787826660] req@ffff976889efe940 x1874670564049664/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826660.575281:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564049664/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826660.575303:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826660.575307:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826660.575311:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564049664:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826660.575334:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564049792:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826660.575355:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826660.575367:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826660/real 1787826660] req@ffff976889eff480 x1874670564049792/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826660.575378:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564049792/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826660.575386:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826660.575389:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826660.575392:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564049792:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826660.575398:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564049920:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826660.575418:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826660.575428:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826660/real 1787826660] req@ffff976889efe580 x1874670564049920/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826660.575439:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564049920/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826660.575446:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826660.575449:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826660.575452:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564049920:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826660.575457:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564050048:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826660.575477:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826660.575488:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826660/real 1787826660] req@ffff976889efcb40 x1874670564050048/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826660.575498:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564050048/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826660.575503:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826660.575505:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826660.575507:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564050048:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826660.575511:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564050176:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826660.575529:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826660.575535:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826660/real 1787826660] req@ffff976889efd2c0 x1874670564050176/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826660.575541:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd2c0 x1874670564050176/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826660.575546:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826660.575560:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826660.575562:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564050176:192.168.203.139@tcp:8:kworker.0 00010000:00010000:1.0:1787826660.831028:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826661.598890:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:0.0:1787826661.854981:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826662.622934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826662.879025:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826663.646940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826663.903125:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826664.670944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:0.0:1787826664.927091:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826665.694939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787826665.695662:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826665.695668:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826665.695672:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826665.695680:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1605 00000100:00080000:0.0:1787826665.695690:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826665.695696:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787826665.695729:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564050304/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826665.695761:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826665.695765:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826665.695766:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826665.695770:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1605 00000100:00080000:0.0:1787826665.695773:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826665.695777:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826665.695787:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826665.695801:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564050432/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826665.695801:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564050304:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787826665.695816:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826665.695819:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826665.695820:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826665.695872:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1605 00000100:00080000:0.0:1787826665.695876:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826665.695881:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826665.695887:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826665.695902:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564050560/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826665.695914:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787826665.695915:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787826665.695921:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826665.695923:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826665.695928:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1605 00000100:00080000:0.0:1787826665.695933:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826665.695937:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826665.695942:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787826665.695954:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564050688/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826665.695965:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826665.695968:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826665.695970:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826665.695974:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1605 00000100:00080000:0.0:1787826665.695977:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826665.695981:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826665.695985:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826665.695994:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564050816/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826665.696054:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826665/real 1787826665] req@ffff976882aee940 x1874670564050304/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826665.696079:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564050304/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826665.696090:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826665.696094:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826665.696098:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564050304:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826665.696119:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564050432:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826665.696143:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826665.696155:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826665/real 1787826665] req@ffff976882aef0c0 x1874670564050432/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826665.696208:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564050432/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826665.696218:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826665.696221:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826665.696225:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564050432:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826665.696230:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564050560:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826665.696248:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826665.696256:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826665/real 1787826665] req@ffff976882aee580 x1874670564050560/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826665.696263:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564050560/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826665.696271:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826665.696273:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826665.696275:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564050560:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826665.696278:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564050688:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826665.696287:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826665.696292:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826665/real 1787826665] req@ffff976882aeed00 x1874670564050688/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826665.696299:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564050688/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826665.696304:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826665.696306:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826665.696309:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564050688:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826665.696314:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564050816:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826665.696331:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826665.696341:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826665/real 1787826665] req@ffff976882aec3c0 x1874670564050816/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826665.696352:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564050816/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826665.696359:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826665.696361:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826665.696365:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564050816:192.168.203.139@tcp:8:kworker.0 00010000:00010000:2.0:1787826665.951042:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826666.718945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826666.975415:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826667.742949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:0.0:1787826667.999020:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826668.767027:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826669.023047:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826669.790928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826670.046993:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826670.814970:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826670.815620:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826670.815626:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826670.815631:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826670.815640:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1610 00000100:00080000:1.0:1787826670.815650:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826670.815656:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826670.815694:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efda40 x1874670564050944/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826670.815718:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826670.815721:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826670.815723:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826670.815726:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1610 00000100:00080000:1.0:1787826670.815729:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826670.815732:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826670.815740:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826670.815754:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564051072/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826670.815764:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826670.815767:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826670.815769:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826670.815773:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1610 00000100:00100000:2.0:1787826670.815774:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564050944:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787826670.815777:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826670.815783:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826670.815789:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826670.815800:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564051200/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826670.815808:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826670.815811:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826670.815813:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826670.815817:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1610 00000100:00080000:1.0:1787826670.815820:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826670.815851:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826670.815856:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826670.815868:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd2c0 x1874670564051328/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826670.815877:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826670.815881:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826670.815883:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787826670.815888:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787826670.815889:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1610 00000100:00080000:1.0:1787826670.815895:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826670.815899:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826670.815903:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787826670.815914:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564051456/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826670.815942:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826670/real 1787826670] req@ffff976889efda40 x1874670564050944/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826670.815960:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efda40 x1874670564050944/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826670.815970:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826670.815974:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826670.815977:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564050944:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826670.816004:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564051072:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826670.816028:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826670.816041:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826670/real 1787826670] req@ffff976889efe1c0 x1874670564051072/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826670.816052:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564051072/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826670.816066:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826670.816069:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826670.816072:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564051072:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826670.816078:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564051200:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826670.816095:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826670.816105:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826670/real 1787826670] req@ffff976889efc780 x1874670564051200/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826670.816116:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564051200/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826670.816123:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826670.816126:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826670.816129:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564051200:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826670.816135:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564051328:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826670.816163:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826670.816175:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826670/real 1787826670] req@ffff976889efd2c0 x1874670564051328/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826670.816187:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd2c0 x1874670564051328/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826670.816195:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826670.816198:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826670.816201:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564051328:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826670.816207:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564051456:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826670.816226:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826670.816237:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826670/real 1787826670] req@ffff976889efcb40 x1874670564051456/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826670.816253:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564051456/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826670.816260:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826670.816275:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826670.816278:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564051456:192.168.203.139@tcp:8:kworker.0 00010000:00010000:0.0:1787826671.071004:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826671.839135:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826672.095008:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826672.863203:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826673.118971:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826673.887652:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:0.0:1787826674.143392:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826674.910966:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826675.167686:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787826675.934925:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787826675.934932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826675.934933:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826675.934938:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826675.934955:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1615 00000100:00080000:1.0:1787826675.934966:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826675.934974:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826675.935013:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564051584/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826675.935037:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826675.935041:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826675.935042:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826675.935047:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1615 00000100:00080000:1.0:1787826675.935050:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826675.935054:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826675.935064:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826675.935069:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564051584:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826675.935079:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564051712/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826675.935090:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826675.935094:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826675.935096:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826675.935099:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1615 00000100:00080000:1.0:1787826675.935102:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826675.935106:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826675.935110:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826675.935123:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564051840/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826675.935133:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826675.935136:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826675.935138:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787826675.935142:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787826675.935145:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1615 00000100:00080000:1.0:1787826675.935151:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826675.935154:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826675.935158:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826675.935182:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd680 x1874670564051968/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826675.935192:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826675.935195:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826675.935197:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787826675.935201:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826675/real 1787826675] req@ffff976889efe580 x1874670564051584/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826675.935204:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1615 00000100:00080000:1.0:1787826675.935212:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826675.935218:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826675.935225:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564051584/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787826675.935228:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787826675.935240:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787826675.935247:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564052096/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826675.935247:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826675.935253:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564051584:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826675.935285:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564051712:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826675.935305:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826675.935316:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826675/real 1787826675] req@ffff976889eff480 x1874670564051712/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826675.935327:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564051712/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826675.935333:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826675.935336:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826675.935338:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564051712:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826675.935343:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564051840:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826675.935358:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826675.935367:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826675/real 1787826675] req@ffff976889efe940 x1874670564051840/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826675.935378:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564051840/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826675.935386:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826675.935389:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826675.935391:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564051840:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826675.935396:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564051968:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826675.935411:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826675.935420:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826675/real 1787826675] req@ffff976889efd680 x1874670564051968/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826675.935429:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd680 x1874670564051968/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826675.935453:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826675.935456:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826675.935459:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564051968:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826675.935465:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564052096:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826675.935483:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826675.935494:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826675/real 1787826675] req@ffff976889eff0c0 x1874670564052096/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826675.935504:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564052096/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826675.935511:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826675.935515:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826675.935518:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564052096:192.168.203.139@tcp:8:kworker.0 00010000:00010000:1.0:1787826676.191026:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826676.960170:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826677.214994:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826677.983196:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826678.239714:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826679.006932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826679.263003:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826680.030936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826680.286978:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787826681.054933:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826681.054943:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826681.054947:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826681.054957:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1620 00000100:00100000:2.0:1787826681.054959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826681.054972:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826681.054982:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826681.055017:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564052224/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826681.055045:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826681.055049:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826681.055051:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826681.055056:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1620 00000100:00080000:1.0:1787826681.055060:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826681.055064:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826681.055074:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826681.055085:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564052224:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826681.055091:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564052352/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826681.055113:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826681.055117:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826681.055119:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826681.055123:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1620 00000100:00080000:1.0:1787826681.055128:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826681.055131:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826681.055137:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787826681.055150:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826681.055153:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564052480/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826681.055165:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826681.055168:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826681.055170:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826681.055175:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1620 00000100:00080000:1.0:1787826681.055186:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826681.055190:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826681.055196:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787826681.055203:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826681/real 1787826681] req@ffff976889effc00 x1874670564052224/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826681.055210:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564052608/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826681.055234:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826681.055239:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826681.055243:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826681.055249:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1620 00000100:00080000:1.0:1787826681.055253:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787826681.055255:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564052224/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826681.055259:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826681.055269:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787826681.055272:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826681.055279:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826681.055284:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd680 x1874670564052736/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787826681.055287:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564052224:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826681.055318:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564052352:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826681.055345:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826681.055359:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826681/real 1787826681] req@ffff976889efed00 x1874670564052352/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826681.055371:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564052352/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826681.055378:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826681.055381:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826681.055385:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564052352:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826681.055390:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564052480:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787826681.055423:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826681.055435:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826681/real 1787826681] req@ffff976889efc3c0 x1874670564052480/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826681.055447:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564052480/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826681.055460:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826681.055463:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826681.055466:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564052480:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826681.055472:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564052608:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826681.055492:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826681.055502:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826681/real 1787826681] req@ffff976889eff0c0 x1874670564052608/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826681.055515:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564052608/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826681.055531:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826681.055535:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826681.055538:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564052608:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826681.055545:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564052736:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787826681.055566:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826681.055584:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826681/real 1787826681] req@ffff976889efd680 x1874670564052736/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826681.055596:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd680 x1874670564052736/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826681.055603:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826681.055607:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787826681.055611:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564052736:192.168.203.139@tcp:8:kworker.0 00010000:00010000:2.0:1787826681.311177:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826682.078955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826682.335774:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826683.102886:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826683.358977:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826684.126920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:0.0:1787826684.383018:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826685.150902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826685.406992:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787826686.175028:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787826686.175035:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787826686.175046:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787826686.175053:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826686.175067:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1626 00000100:00080000:2.0:1787826686.175092:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826686.175100:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826686.175146:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9a40 x1874670564052864/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826686.175180:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826686.175184:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826686.175186:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826686.175191:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1626 00000100:00080000:2.0:1787826686.175194:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826686.175198:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826686.175210:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826686.175223:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564052992/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826686.175233:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826686.175237:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787826686.175240:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564052864:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787826686.175241:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826686.175248:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1626 00000100:00080000:2.0:1787826686.175253:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826686.175256:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826686.175261:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826686.175273:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564053120/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826686.175282:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826686.175286:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826686.175288:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826686.175292:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1626 00000100:00080000:2.0:1787826686.175296:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826686.175300:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826686.175305:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787826686.175317:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564053248/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787826686.175326:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787826686.175329:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826686.175333:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826686.175335:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826686.175340:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1626 00000100:00080000:2.0:1787826686.175344:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826686.175348:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826686.175354:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787826686.175366:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564053376/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826686.175396:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826686/real 1787826686] req@ffff976889cf9a40 x1874670564052864/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826686.175417:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9a40 x1874670564052864/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826686.175434:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826686.175438:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826686.175443:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564052864:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826686.175468:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564052992:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826686.175488:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826686.175498:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826686/real 1787826686] req@ffff976889cfb0c0 x1874670564052992/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826686.175506:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564052992/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826686.175511:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826686.175513:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826686.175516:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564052992:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826686.175519:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564053120:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826686.175531:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826686.175537:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826686/real 1787826686] req@ffff976889cf9e00 x1874670564053120/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826686.175544:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564053120/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826686.175549:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826686.175551:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826686.175553:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564053120:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826686.175559:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564053248:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826686.175577:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826686.175587:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826686/real 1787826686] req@ffff976889cfa940 x1874670564053248/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826686.175598:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564053248/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826686.175606:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826686.175614:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826686.175618:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564053248:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826686.175624:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564053376:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826686.175642:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826686.175649:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826686/real 1787826686] req@ffff976889cf8f00 x1874670564053376/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826686.175656:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564053376/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826686.175662:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826686.175665:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826686.175668:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564053376:192.168.203.139@tcp:8:kworker.0 00010000:00010000:3.0:1787826686.430953:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826687.198899:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:0.0:1787826687.454960:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826688.222930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826688.479001:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826689.247547:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826689.502946:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826690.270939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:0.0:1787826690.527002:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787826691.294934:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787826691.294937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787826691.294946:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787826691.294951:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826691.294961:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1631 00000100:00080000:2.0:1787826691.294971:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826691.294978:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826691.295016:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564053504/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826691.295034:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826691.295038:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826691.295040:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826691.295044:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1631 00000100:00080000:2.0:1787826691.295048:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826691.295052:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787826691.295058:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564053504:192.168.203.139@tcp:250:kworker.0 00000008:00100000:2.0:1787826691.295064:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826691.295079:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564053632/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826691.295088:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826691.295090:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787826691.295092:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826691.295095:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1631 00000100:00080000:2.0:1787826691.295099:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826691.295103:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826691.295108:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826691.295120:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564053760/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826691.295128:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787826691.295133:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787826691.295135:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826691.295138:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826691.295157:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1631 00000100:00080000:2.0:1787826691.295162:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826691.295166:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826691.295172:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787826691.295174:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826691/real 1787826691] req@ffff976889cf8000 x1874670564053504/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826691.295186:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564053888/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826691.295194:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564053504/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826691.295198:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826691.295206:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826691.295210:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826691.295215:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826691.295216:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1631 00000100:00080000:2.0:1787826691.295223:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826691.295224:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:2.0:1787826691.295229:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787826691.295232:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564053504:192.168.203.139@tcp:250:kworker.0 00000008:00100000:2.0:1787826691.295236:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787826691.295248:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564054016/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826691.295259:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564053632:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826691.295282:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826691.295295:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826691/real 1787826691] req@ffff976889cfbc00 x1874670564053632/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826691.295306:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564053632/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826691.295313:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826691.295316:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826691.295319:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564053632:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826691.295324:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564053760:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826691.295354:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826691.295365:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826691/real 1787826691] req@ffff976889cf8f00 x1874670564053760/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826691.295382:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564053760/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826691.295390:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826691.295393:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826691.295396:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564053760:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826691.295401:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564053888:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826691.295418:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826691.295427:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826691/real 1787826691] req@ffff976889cfa940 x1874670564053888/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826691.295437:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564053888/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826691.295445:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826691.295448:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826691.295451:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564053888:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826691.295465:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564054016:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826691.295488:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826691.295498:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826691/real 1787826691] req@ffff976889cf9e00 x1874670564054016/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826691.295508:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564054016/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826691.295521:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826691.295524:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826691.295527:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564054016:192.168.203.139@tcp:8:kworker.0 00010000:00010000:1.0:1787826691.550968:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826692.318885:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826692.575046:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826693.342925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:0.0:1787826693.599061:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826694.367045:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826694.622989:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826695.391000:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826695.646953:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826696.414893:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826696.414927:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826696.414934:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826696.414938:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826696.414946:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1636 00000100:00080000:1.0:1787826696.414955:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826696.414962:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826696.414997:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564054144/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826696.415022:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826696.415026:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826696.415028:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826696.415032:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1636 00000100:00080000:1.0:1787826696.415035:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826696.415040:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826696.415049:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826696.415059:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564054144:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826696.415065:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564054272/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826696.415076:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826696.415079:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826696.415081:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826696.415085:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1636 00000100:00080000:1.0:1787826696.415088:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826696.415092:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826696.415096:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826696.415109:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564054400/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787826696.415118:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787826696.415118:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826696.415124:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826696.415126:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826696.415132:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1636 00000100:00080000:1.0:1787826696.415136:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826696.415140:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826696.415145:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787826696.415151:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826696/real 1787826696] req@ffff976889efe940 x1874670564054144/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826696.415166:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564054528/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826696.415170:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564054144/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826696.415185:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826696.415190:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826696.415192:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826696.415193:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826696.415200:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:1.0:1787826696.415201:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1636 00000100:00080000:1.0:1787826696.415207:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787826696.415209:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564054144:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787826696.415213:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826696.415220:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787826696.415231:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd2c0 x1874670564054656/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826696.415237:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564054272:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826696.415265:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826696.415296:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826696/real 1787826696] req@ffff976889eff480 x1874670564054272/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826696.415320:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564054272/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826696.415327:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826696.415330:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826696.415334:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564054272:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826696.415340:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564054400:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826696.415358:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826696.415369:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826696/real 1787826696] req@ffff976889efe580 x1874670564054400/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826696.415379:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564054400/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826696.415384:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826696.415386:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826696.415388:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564054400:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826696.415392:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564054528:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826696.415405:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826696.415411:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826696/real 1787826696] req@ffff976889efcb40 x1874670564054528/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826696.415418:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564054528/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826696.415426:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826696.415429:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826696.415432:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564054528:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826696.415447:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564054656:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826696.415478:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826696.415488:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826696/real 1787826696] req@ffff976889efd2c0 x1874670564054656/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826696.415496:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd2c0 x1874670564054656/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826696.415501:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826696.415502:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826696.415504:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564054656:192.168.203.139@tcp:8:kworker.0 00010000:00010000:0.0:1787826696.670995:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826697.439009:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826697.694987:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826698.462986:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826698.719020:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826699.486951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:0.0:1787826699.743178:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826700.510893:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826700.767063:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787826701.534919:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787826701.534921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826701.534928:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826701.534933:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826701.534940:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1641 00000100:00080000:1.0:1787826701.534949:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826701.534966:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826701.535001:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564054784/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826701.535021:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826701.535023:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826701.535025:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826701.535027:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1641 00000100:00080000:1.0:1787826701.535030:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826701.535032:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826701.535039:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826701.535046:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564054912/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826701.535053:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826701.535055:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826701.535056:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826701.535058:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1641 00000100:00080000:1.0:1787826701.535060:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826701.535062:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826701.535064:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826701.535070:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564054784:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826701.535071:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efda40 x1874670564055040/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826701.535083:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826701.535087:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826701.535088:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826701.535092:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1641 00000100:00080000:1.0:1787826701.535096:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826701.535099:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826701.535104:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826701.535117:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd2c0 x1874670564055168/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787826701.535138:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787826701.535146:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826701.535150:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826701.535152:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826701.535156:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1641 00000100:00080000:1.0:1787826701.535161:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826701.535165:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826701.535170:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826701.535181:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826701/real 1787826701] req@ffff976889efc780 x1874670564054784/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826701.535185:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564055296/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826701.535214:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564054784/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826701.535225:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826701.535230:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826701.535233:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564054784:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826701.535240:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564054912:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826701.535275:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826701.535290:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826701/real 1787826701] req@ffff976889efe1c0 x1874670564054912/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826701.535315:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564054912/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826701.535323:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826701.535327:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826701.535330:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564054912:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826701.535353:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564055040:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826701.535373:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826701.535385:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826701/real 1787826701] req@ffff976889efda40 x1874670564055040/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826701.535397:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efda40 x1874670564055040/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826701.535405:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826701.535408:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826701.535411:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564055040:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826701.535417:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564055168:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826701.535437:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826701.535447:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826701/real 1787826701] req@ffff976889efd2c0 x1874670564055168/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826701.535457:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd2c0 x1874670564055168/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826701.535470:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826701.535473:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826701.535476:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564055168:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826701.535482:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564055296:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826701.535500:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826701.535511:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826701/real 1787826701] req@ffff976889efcb40 x1874670564055296/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826701.535522:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564055296/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826701.535529:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826701.535533:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826701.535536:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564055296:192.168.203.139@tcp:8:kworker.0 00010000:00010000:3.0:1787826701.790970:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826702.558936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826702.814998:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826703.582958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826703.839029:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826704.606938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826704.862938:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826705.630940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826705.887007:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826706.654982:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787826706.655168:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826706.655174:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787826706.655178:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826706.655187:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1646 00000100:00080000:2.0:1787826706.655198:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826706.655206:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826706.655263:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564055424/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826706.655289:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826706.655293:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826706.655295:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826706.655299:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1646 00000100:00080000:2.0:1787826706.655305:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826706.655309:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826706.655321:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826706.655339:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9a40 x1874670564055552/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826706.655354:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564055424:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787826706.655358:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826706.655371:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787826706.655373:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826706.655379:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1646 00000100:00080000:2.0:1787826706.655392:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826706.655396:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826706.655401:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826706.655415:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564055680/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826706.655425:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826706.655428:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826706.655430:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826706.655434:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1646 00000100:00080000:2.0:1787826706.655439:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826706.655444:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826706.655449:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1787826706.655460:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826706.655463:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564055808/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826706.655474:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826706.655478:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826706.655480:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826706.655484:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1646 00000100:00080000:2.0:1787826706.655489:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826706.655493:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826706.655499:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787826706.655510:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564055936/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826706.655522:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826706/real 1787826706] req@ffff976889cfb0c0 x1874670564055424/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826706.655546:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564055424/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826706.655558:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826706.655564:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826706.655568:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564055424:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826706.655610:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564055552:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826706.655638:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826706.655653:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826706/real 1787826706] req@ffff976889cf9a40 x1874670564055552/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826706.655667:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9a40 x1874670564055552/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826706.655675:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826706.655678:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826706.655682:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564055552:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826706.655689:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564055680:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826706.655711:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826706.655723:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826706/real 1787826706] req@ffff976889cfb480 x1874670564055680/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826706.655735:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564055680/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826706.655744:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826706.655747:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826706.655750:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564055680:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826706.655757:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564055808:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826706.655778:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826706.655788:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826706/real 1787826706] req@ffff976889cfa580 x1874670564055808/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826706.655801:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564055808/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826706.655809:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826706.655812:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826706.655816:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564055808:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826706.655822:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564055936:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826706.655870:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826706.655880:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826706/real 1787826706] req@ffff976889cfa1c0 x1874670564055936/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826706.655891:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564055936/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826706.655899:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826706.655902:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826706.655904:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564055936:192.168.203.139@tcp:8:kworker.0 00010000:00010000:1.0:1787826706.910957:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826707.678932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826707.934957:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826708.702969:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826708.958972:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826709.726967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826709.982993:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826710.751135:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826711.006995:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826711.774964:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787826711.775246:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826711.775252:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826711.775256:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826711.775263:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1651 00000100:00080000:3.0:1787826711.775273:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826711.775280:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787826711.775312:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564056064/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826711.775339:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826711.775342:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826711.775344:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826711.775348:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1651 00000100:00080000:3.0:1787826711.775351:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826711.775355:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826711.775365:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826711.775377:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564056192/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826711.775381:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564056064:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787826711.775399:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826711.775403:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826711.775405:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826711.775410:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1651 00000100:00080000:3.0:1787826711.775414:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826711.775418:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826711.775423:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826711.775434:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564056320/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826711.775443:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826711.775446:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826711.775448:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826711.775452:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1651 00000100:00080000:3.0:1787826711.775456:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826711.775460:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826711.775464:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787826711.775477:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564056448/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826711.775487:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787826711.775489:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564056192:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787826711.775492:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826711.775496:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826711.775502:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1651 00000100:00080000:3.0:1787826711.775506:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826711.775510:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826711.775515:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826711.775527:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564056320:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787826711.775529:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564056576/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826711.775558:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564056448:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826711.775579:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564056576:192.168.203.139@tcp:8:kworker.0 00010000:00010000:2.0:1787826712.031024:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826712.798961:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826713.055024:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826713.822953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826714.078972:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826714.846976:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:1.0:1787826714.847092:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:1.0:1787826714.847126:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787826714.847139:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787826714.847191:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787826714.847204:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787826714.847214:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787826714.847223:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:0.0:1787826714.847304:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826711/real 1787826714] req@ffff97688cb5c780 x1874670564056448/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826714.847337:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564056448/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826714.847351:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826714.847357:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826714.847362:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564056448:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826714.847371:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826711/real 1787826714] req@ffff97688cb5e1c0 x1874670564056576/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826714.847385:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564056576/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826714.847394:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826714.847398:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826714.847401:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564056576:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826714.847409:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826711/real 1787826714] req@ffff97688cb5c3c0 x1874670564056192/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826714.847422:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564056192/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826714.847430:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826714.847434:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826714.847438:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564056192:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826714.847446:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826711/real 1787826714] req@ffff97688cb5f840 x1874670564056320/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826714.847458:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564056320/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826714.847465:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826714.847469:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826714.847472:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564056320:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826714.847479:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826711/real 1787826714] req@ffff97688cb5cf00 x1874670564056064/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826714.847493:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564056064/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826714.847500:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826714.847503:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826714.847507:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564056064:192.168.203.139@tcp:250:kworker.0 00010000:00010000:2.0:1787826715.103144:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826715.871066:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826716.127082:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826716.895001:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826716.895010:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826716.895014:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826716.895022:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1656 00000100:00100000:3.0:1787826716.895025:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787826716.895031:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826716.895039:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787826716.895076:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564056704/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826716.895108:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826716.895112:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826716.895114:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826716.895118:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1656 00000100:00080000:0.0:1787826716.895122:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826716.895127:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826716.895137:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826716.895150:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef840 x1874670564056832/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826716.895171:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787826716.895174:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564056704:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787826716.895176:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826716.895183:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826716.895190:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1656 00000100:00080000:0.0:1787826716.895194:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826716.895199:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826716.895205:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826716.895221:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564056960/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826716.895231:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826716.895234:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826716.895236:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826716.895240:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1656 00000100:00080000:0.0:1787826716.895246:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826716.895250:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826716.895255:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787826716.895268:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564057088/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826716.895277:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787826716.895283:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787826716.895286:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826716.895292:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826716.895298:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1656 00000100:00080000:0.0:1787826716.895304:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826716.895308:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826716.895315:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826716.895330:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564057216/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787826716.895354:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826716/real 1787826716] req@ffff976882aed680 x1874670564056704/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826716.895377:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564056704/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826716.895389:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826716.895393:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826716.895398:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564056704:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787826716.895428:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564056832:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787826716.895455:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826716.895469:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826716/real 1787826716] req@ffff976882aef840 x1874670564056832/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826716.895480:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef840 x1874670564056832/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826716.895495:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826716.895499:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826716.895502:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564056832:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787826716.895509:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564056960:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787826716.895532:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826716.895544:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826716/real 1787826716] req@ffff976882aec3c0 x1874670564056960/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826716.895556:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564056960/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826716.895564:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826716.895567:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826716.895570:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564056960:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787826716.895577:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564057088:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787826716.895599:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826716.895610:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826716/real 1787826716] req@ffff976882aeed00 x1874670564057088/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826716.895623:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564057088/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826716.895632:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826716.895635:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826716.895639:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564057088:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787826716.895646:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564057216:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787826716.895665:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826716.895677:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826716/real 1787826716] req@ffff976882aee580 x1874670564057216/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826716.895689:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564057216/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826716.895697:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826716.895701:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826716.895704:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564057216:192.168.203.139@tcp:8:kworker.0 00010000:00010000:3.0:1787826717.150962:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826717.919523:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826718.175121:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826718.942936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826719.198999:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826719.966999:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826720.222999:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826720.990964:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826721.247007:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787826722.015146:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826722.015159:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1787826722.015163:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787826722.015165:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826722.015240:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1661 00000100:00080000:2.0:1787826722.015254:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826722.015263:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826722.015317:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564057344/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826722.015355:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826722.015359:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826722.015361:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826722.015366:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1661 00000100:00080000:2.0:1787826722.015370:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826722.015375:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826722.015387:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826722.015401:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564057472/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826722.015411:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826722.015415:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787826722.015417:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826722.015421:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1661 00000100:00080000:2.0:1787826722.015424:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826722.015428:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826722.015433:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826722.015445:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564057600/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826722.015455:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826722.015458:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826722.015459:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826722.015464:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1661 00000100:00080000:2.0:1787826722.015468:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826722.015472:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826722.015477:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787826722.015488:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564057728/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826722.015496:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826722.015499:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826722.015501:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826722.015504:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1661 00000100:00080000:2.0:1787826722.015508:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826722.015511:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826722.015516:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787826722.015526:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564057856/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787826722.015620:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564057344:192.168.203.139@tcp:250:kworker.0 00000800:00000100:3.0:1787826722.015706:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826722.015769:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826722/real 1787826722] req@ffff976889cfad00 x1874670564057344/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826722.015788:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564057344/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826722.015799:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826722.015804:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826722.015808:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564057344:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787826722.015815:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564057472:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787826722.015863:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826722.015876:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826722/real 1787826722] req@ffff976889cf92c0 x1874670564057472/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826722.015887:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564057472/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826722.015895:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826722.015898:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826722.015901:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564057472:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787826722.015907:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564057600:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787826722.015925:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826722.015935:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826722/real 1787826722] req@ffff976889cfa1c0 x1874670564057600/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826722.015947:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564057600/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826722.015955:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826722.015958:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826722.015961:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564057600:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787826722.015966:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564057728:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787826722.015987:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826722.015998:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826722/real 1787826722] req@ffff976889cfa580 x1874670564057728/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826722.016009:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564057728/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826722.016016:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826722.016020:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826722.016023:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564057728:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787826722.016028:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564057856:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787826722.016047:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826722.016056:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826722/real 1787826722] req@ffff976889cfb480 x1874670564057856/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826722.016067:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564057856/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826722.016075:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826722.016079:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826722.016082:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564057856:192.168.203.139@tcp:8:kworker.0 00010000:00010000:1.0:1787826722.271567:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826723.038946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826723.295061:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826724.062955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826724.319007:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826725.086963:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826725.342994:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826726.110946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826726.366966:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826727.134945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787826727.134949:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826727.134958:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787826727.134963:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826727.134974:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1667 00000100:00080000:2.0:1787826727.134985:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826727.134991:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826727.135031:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9a40 x1874670564057984/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826727.135060:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826727.135062:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826727.135063:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826727.135067:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1667 00000100:00080000:2.0:1787826727.135069:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826727.135071:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826727.135081:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826727.135105:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564058112/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787826727.135112:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564057984:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787826727.135117:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826727.135124:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787826727.135125:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826727.135129:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1667 00000100:00080000:2.0:1787826727.135131:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826727.135134:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826727.135138:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826727.135147:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564058240/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826727.135153:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826727.135156:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826727.135157:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826727.135161:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1667 00000100:00080000:2.0:1787826727.135176:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826727.135179:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826727.135183:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787826727.135199:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564058368/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787826727.135200:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787826727.135212:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826727.135215:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826727.135217:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826727.135221:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1667 00000100:00080000:2.0:1787826727.135223:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826727.135226:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826727.135229:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787826727.135239:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564058496/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787826727.135265:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826727/real 1787826727] req@ffff976889cf9a40 x1874670564057984/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826727.135284:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9a40 x1874670564057984/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826727.135295:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826727.135299:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826727.135304:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564057984:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787826727.135327:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564058112:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787826727.135351:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826727.135363:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826727/real 1787826727] req@ffff976889cfb0c0 x1874670564058112/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826727.135375:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564058112/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826727.135383:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826727.135386:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826727.135389:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564058112:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787826727.135396:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564058240:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787826727.135413:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826727.135420:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826727/real 1787826727] req@ffff976889cf9e00 x1874670564058240/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826727.135428:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564058240/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826727.135433:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826727.135435:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826727.135437:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564058240:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787826727.135441:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564058368:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787826727.135452:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826727.135458:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826727/real 1787826727] req@ffff976889cfa940 x1874670564058368/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826727.135464:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564058368/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826727.135469:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826727.135471:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826727.135472:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564058368:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787826727.135475:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564058496:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787826727.135484:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826727.135489:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826727/real 1787826727] req@ffff976889cf8f00 x1874670564058496/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826727.135494:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564058496/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826727.135503:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826727.135505:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826727.135506:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564058496:192.168.203.139@tcp:8:kworker.0 00010000:00010000:2.0:1787826727.391008:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826728.158925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826728.414984:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826729.182946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826729.438970:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826730.206932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826730.462964:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826731.230921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826731.486989:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787826732.254932:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787826732.254933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787826732.254949:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787826732.254954:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826732.254964:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1672 00000100:00080000:2.0:1787826732.254973:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826732.254980:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826732.255021:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564058624/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826732.255061:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826732.255064:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826732.255066:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826732.255070:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1672 00000100:00080000:2.0:1787826732.255074:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826732.255078:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826732.255087:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826732.255093:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564058624:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826732.255100:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564058752/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826732.255110:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826732.255113:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787826732.255115:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826732.255118:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1672 00000100:00080000:2.0:1787826732.255121:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826732.255125:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826732.255129:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826732.255140:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564058880/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826732.255148:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826732.255150:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826732.255152:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826732.255158:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1672 00000800:00000100:3.0:1787826732.255158:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787826732.255165:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826732.255178:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826732.255185:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787826732.255196:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564059008/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826732.255204:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826732.255206:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826732.255208:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826732.255210:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1672 00000100:00080000:2.0:1787826732.255213:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826732.255216:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826732.255220:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826732.255227:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826732/real 1787826732] req@ffff976889cfbc00 x1874670564058624/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826732.255231:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564059136/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787826732.255265:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564058624/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826732.255276:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826732.255280:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826732.255284:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564058624:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787826732.255305:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564058752:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787826732.255326:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826732.255338:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826732/real 1787826732] req@ffff976889cf8000 x1874670564058752/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826732.255355:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564058752/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826732.255362:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826732.255364:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826732.255367:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564058752:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787826732.255372:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564058880:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787826732.255388:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826732.255397:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826732/real 1787826732] req@ffff976889cf8f00 x1874670564058880/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826732.255408:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564058880/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826732.255414:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826732.255417:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826732.255420:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564058880:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787826732.255425:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564059008:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787826732.255440:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826732.255449:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826732/real 1787826732] req@ffff976889cfa940 x1874670564059008/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826732.255460:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564059008/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826732.255473:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826732.255476:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826732.255479:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564059008:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787826732.255484:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564059136:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787826732.255499:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826732.255508:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826732/real 1787826732] req@ffff976889cf9e00 x1874670564059136/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826732.255517:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564059136/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826732.255524:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826732.255526:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826732.255529:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564059136:192.168.203.139@tcp:8:kworker.0 00010000:00010000:3.0:1787826732.510950:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826733.278963:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826733.535011:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826734.304088:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826734.559104:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826735.327045:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826735.583049:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826736.351658:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826736.607057:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826737.374934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787826737.375914:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826737.375919:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826737.375923:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826737.375931:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1677 00000100:00080000:0.0:1787826737.375941:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826737.375948:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787826737.375981:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564059264/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826737.376010:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826737.376026:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826737.376027:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826737.376030:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1677 00000100:00080000:0.0:1787826737.376034:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826737.376037:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826737.376045:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826737.376056:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564059392/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826737.376064:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826737.376067:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826737.376068:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826737.376071:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1677 00000100:00080000:0.0:1787826737.376073:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826737.376076:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826737.376079:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826737.376093:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564059520/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826737.376100:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826737.376103:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826737.376105:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826737.376108:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1677 00000100:00080000:0.0:1787826737.376111:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826737.376114:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826737.376118:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787826737.376127:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564059648/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826737.376135:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826737.376138:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826737.376139:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826737.376142:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1677 00000100:00080000:0.0:1787826737.376145:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826737.376147:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826737.376151:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826737.376158:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564059776/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787826737.376227:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564059264:192.168.203.139@tcp:250:kworker.0 00000800:00000100:3.0:1787826737.376282:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826737.376327:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826737/real 1787826737] req@ffff976882aef0c0 x1874670564059264/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826737.376349:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564059264/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826737.376358:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826737.376448:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826737.376453:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564059264:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787826737.376460:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564059392:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787826737.376549:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826737.376563:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826737/real 1787826737] req@ffff976882aee940 x1874670564059392/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826737.376575:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564059392/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826737.376583:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826737.376585:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826737.376588:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564059392:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787826737.376594:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564059520:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787826737.376613:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826737.376624:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826737/real 1787826737] req@ffff976882aee580 x1874670564059520/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826737.376635:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564059520/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826737.376647:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826737.376650:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826737.376652:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564059520:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787826737.376658:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564059648:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787826737.376678:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826737.376687:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826737/real 1787826737] req@ffff976882aeed00 x1874670564059648/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826737.376697:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564059648/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826737.376704:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826737.376707:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826737.376710:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564059648:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787826737.376715:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564059776:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787826737.376734:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826737.376744:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826737/real 1787826737] req@ffff976882aec3c0 x1874670564059776/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826737.376754:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564059776/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826737.376761:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826737.376763:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826737.376765:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564059776:192.168.203.139@tcp:8:kworker.0 00010000:00010000:1.0:1787826737.631503:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826738.400271:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826738.655119:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826739.422945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826739.679037:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826740.446942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826740.703662:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826741.470923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826741.727148:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826742.494954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787826742.494956:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826742.494974:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826742.494988:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826742.494998:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1682 00000100:00080000:3.0:1787826742.495009:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826742.495016:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787826742.495057:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564059904/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826742.495086:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826742.495089:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826742.495091:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826742.495095:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1682 00000100:00080000:3.0:1787826742.495099:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826742.495103:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826742.495113:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826742.495129:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564060032/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826742.495136:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564059904:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787826742.495143:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826742.495148:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826742.495150:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826742.495155:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1682 00000100:00080000:3.0:1787826742.495160:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826742.495164:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826742.495169:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826742.495182:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564060160/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826742.495192:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826742.495195:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826742.495197:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826742.495201:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1682 00000100:00080000:3.0:1787826742.495205:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826742.495209:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:0.0:1787826742.495215:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:3.0:1787826742.495217:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787826742.495230:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564060288/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826742.495239:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826742.495241:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826742.495242:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826742.495245:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1682 00000100:00080000:3.0:1787826742.495248:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826742.495251:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826742.495255:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826742.495263:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564060416/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826742.495274:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826742/real 1787826742] req@ffff97688cb5d2c0 x1874670564059904/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826742.495301:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564059904/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826742.495312:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826742.495317:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826742.495321:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564059904:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826742.495343:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564060032:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826742.495361:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826742.495371:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826742/real 1787826742] req@ffff97688cb5cf00 x1874670564060032/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826742.495387:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564060032/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826742.495393:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826742.495395:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826742.495397:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564060032:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826742.495401:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564060160:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826742.495412:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826742.495418:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826742/real 1787826742] req@ffff97688cb5f840 x1874670564060160/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826742.495425:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564060160/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826742.495430:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826742.495432:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826742.495434:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564060160:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826742.495436:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564060288:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826742.495446:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826742.495452:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826742/real 1787826742] req@ffff97688cb5c3c0 x1874670564060288/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826742.495459:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564060288/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826742.495463:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826742.495465:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826742.495466:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564060288:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826742.495469:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564060416:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826742.495478:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826742.495483:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826742/real 1787826742] req@ffff97688cb5e1c0 x1874670564060416/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826742.495489:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564060416/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826742.495494:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826742.495495:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826742.495497:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564060416:192.168.203.139@tcp:8:kworker.0 00010000:00010000:2.0:1787826742.751019:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826743.519000:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826743.774936:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826744.543018:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826744.799042:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826745.566951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826745.823174:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826746.590985:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826746.847261:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826747.614990:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787826747.615486:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826747.615493:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787826747.615498:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826747.615509:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1687 00000100:00080000:2.0:1787826747.615523:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826747.615532:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826747.615586:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564060544/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826747.615625:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826747.615629:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826747.615631:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826747.615636:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1687 00000100:00080000:2.0:1787826747.615639:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826747.615644:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826747.615656:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826747.615669:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9a40 x1874670564060672/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826747.615678:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826747.615681:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787826747.615683:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826747.615687:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1687 00000100:00080000:2.0:1787826747.615691:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826747.615695:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826747.615700:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826747.615713:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564060800/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826747.615719:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564060544:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787826747.615727:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826747.615731:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826747.615733:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826747.615738:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1687 00000100:00080000:2.0:1787826747.615742:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826747.615746:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826747.615752:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787826747.615765:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564060928/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826747.615774:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826747.615777:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826747.615779:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826747.615783:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1687 00000100:00080000:2.0:1787826747.615787:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826747.615790:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826747.615795:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787826747.615800:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826747.615809:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564061056/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826747.615890:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826747/real 1787826747] req@ffff976889cfb0c0 x1874670564060544/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826747.615910:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564060544/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826747.615922:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826747.615927:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826747.615931:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564060544:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826747.615939:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564060672:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826747.615976:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826747.615990:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826747/real 1787826747] req@ffff976889cf9a40 x1874670564060672/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826747.616008:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9a40 x1874670564060672/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826747.616016:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826747.616020:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826747.616023:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564060672:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826747.616050:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564060800:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826747.616069:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826747.616090:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826747/real 1787826747] req@ffff976889cfb480 x1874670564060800/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826747.616112:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564060800/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826747.616121:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826747.616124:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826747.616127:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564060800:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826747.616133:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564060928:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826747.616152:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826747.616164:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826747/real 1787826747] req@ffff976889cfa580 x1874670564060928/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826747.616175:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564060928/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826747.616182:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826747.616186:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826747.616189:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564060928:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826747.616195:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564061056:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826747.616214:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826747.616224:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826747/real 1787826747] req@ffff976889cfa1c0 x1874670564061056/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826747.616236:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564061056/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826747.616249:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826747.616253:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826747.616256:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564061056:192.168.203.139@tcp:8:kworker.0 00010000:00010000:3.0:1787826747.871091:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826748.639038:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826748.895045:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826749.662950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826749.919006:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826750.686945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826750.942965:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826751.710949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826751.967010:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787826752.734932:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826752.734942:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1787826752.734942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787826752.734946:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826752.734955:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1692 00000100:00080000:2.0:1787826752.734964:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826752.734973:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826752.735008:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564061184/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826752.735031:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826752.735035:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826752.735037:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826752.735041:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1692 00000100:00080000:2.0:1787826752.735045:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826752.735048:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787826752.735057:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564061184:192.168.203.139@tcp:250:kworker.0 00000008:00100000:2.0:1787826752.735057:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826752.735074:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564061312/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826752.735096:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826752.735099:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787826752.735101:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826752.735105:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1692 00000100:00080000:2.0:1787826752.735110:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826752.735114:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:0.0:1787826752.735118:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:2.0:1787826752.735122:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826752.735134:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564061440/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826752.735142:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826752.735144:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826752.735146:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787826752.735149:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826752/real 1787826752] req@ffff976889cf92c0 x1874670564061184/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826752.735152:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1692 00000100:00080000:2.0:1787826752.735159:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826752.735164:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787826752.735176:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564061184/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787826752.735177:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00080000:0.0:1787826752.735185:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826752.735189:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826752.735192:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564061184:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826752.735197:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564061568/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826752.735207:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826752.735210:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787826752.735212:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564061312:192.168.203.139@tcp:38:kworker.0 00000100:00080000:2.0:1787826752.735213:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826752.735220:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1692 00000100:00080000:2.0:1787826752.735225:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826752.735229:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:0.0:1787826752.735232:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:2.0:1787826752.735237:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826752.735245:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826752/real 1787826752] req@ffff976889cfad00 x1874670564061312/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826752.735253:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564061696/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826752.735257:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564061312/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826752.735281:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826752.735285:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826752.735290:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564061312:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826752.735297:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564061440:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826752.735317:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826752.735328:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826752/real 1787826752] req@ffff976889cfa1c0 x1874670564061440/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826752.735341:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564061440/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826752.735349:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826752.735353:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826752.735356:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564061440:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826752.735362:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564061568:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826752.735382:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826752.735395:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826752/real 1787826752] req@ffff976889cfa580 x1874670564061568/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826752.735409:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564061568/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826752.735418:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826752.735421:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826752.735425:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564061568:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826752.735450:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564061696:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826752.735472:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826752.735484:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826752/real 1787826752] req@ffff976889cfb480 x1874670564061696/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826752.735495:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564061696/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826752.735502:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826752.735505:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826752.735508:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564061696:192.168.203.139@tcp:8:kworker.0 00010000:00010000:1.0:1787826752.991033:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826753.758940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826754.015033:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826754.782938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826755.038971:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826755.806945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826756.063014:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826756.830931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826757.086980:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826757.854938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787826757.854953:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826757.854959:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787826757.854963:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826757.854971:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1697 00000100:00080000:2.0:1787826757.854980:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826757.854986:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826757.855018:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9a40 x1874670564061824/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826757.855042:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826757.855046:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826757.855048:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826757.855051:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1697 00000100:00080000:2.0:1787826757.855055:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826757.855059:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826757.855069:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826757.855082:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564061952/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826757.855093:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564061824:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787826757.855093:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826757.855101:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787826757.855104:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826757.855109:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1697 00000100:00080000:2.0:1787826757.855112:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826757.855116:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826757.855122:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826757.855134:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564062080/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826757.855141:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826757.855143:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826757.855144:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826757.855147:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1697 00000100:00080000:2.0:1787826757.855150:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826757.855152:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826757.855156:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1787826757.855161:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826757.855166:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564062208/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826757.855173:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826757.855175:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826757.855176:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826757.855178:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1697 00000100:00080000:2.0:1787826757.855181:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826757.855184:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826757.855188:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787826757.855197:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564062336/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826757.855214:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826757/real 1787826757] req@ffff976889cf9a40 x1874670564061824/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826757.855233:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9a40 x1874670564061824/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826757.855243:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826757.855247:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826757.855252:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564061824:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826757.855272:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564061952:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826757.855294:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826757.855307:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826757/real 1787826757] req@ffff976889cfb0c0 x1874670564061952/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826757.855319:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564061952/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826757.855328:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826757.855331:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826757.855334:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564061952:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826757.855340:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564062080:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826757.855358:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826757.855369:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826757/real 1787826757] req@ffff976889cf9e00 x1874670564062080/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826757.855380:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564062080/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826757.855388:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826757.855391:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826757.855394:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564062080:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826757.855399:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564062208:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826757.855417:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826757.855427:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826757/real 1787826757] req@ffff976889cfa940 x1874670564062208/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826757.855448:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564062208/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826757.855456:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826757.855460:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826757.855463:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564062208:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826757.855468:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564062336:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826757.855485:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826757.855496:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826757/real 1787826757] req@ffff976889cf8f00 x1874670564062336/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826757.855515:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564062336/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826757.855522:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826757.855526:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826757.855529:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564062336:192.168.203.139@tcp:8:kworker.0 00010000:00010000:2.0:1787826758.111098:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826758.878926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826759.135530:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826759.902950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826760.158933:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826760.926914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826761.182941:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826761.950992:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826762.207003:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787826762.974905:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826762.974915:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787826762.974920:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826762.974930:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1702 00000100:00100000:3.0:1787826762.974930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787826762.974942:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826762.974951:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826762.974985:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564062464/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826762.975014:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826762.975018:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826762.975020:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826762.975024:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1702 00000100:00080000:2.0:1787826762.975029:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826762.975032:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826762.975040:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826762.975049:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564062592/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826762.975057:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826762.975059:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787826762.975061:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787826762.975064:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564062464:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787826762.975065:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1702 00000100:00080000:2.0:1787826762.975069:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826762.975074:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826762.975078:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826762.975090:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564062720/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826762.975098:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826762.975102:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826762.975104:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826762.975108:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1702 00000100:00080000:2.0:1787826762.975112:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826762.975116:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826762.975122:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1787826762.975132:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826762.975136:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564062848/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826762.975144:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826762.975146:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826762.975148:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826762.975150:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1702 00000100:00080000:2.0:1787826762.975153:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826762.975156:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826762.975160:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787826762.975171:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564062976/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826762.975196:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826762/real 1787826762] req@ffff976889cf8000 x1874670564062464/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826762.975220:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564062464/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826762.975231:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826762.975242:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826762.975246:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564062464:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826762.975268:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564062592:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826762.975291:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826762.975314:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826762/real 1787826762] req@ffff976889cfbc00 x1874670564062592/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826762.975327:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564062592/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826762.975335:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826762.975338:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826762.975341:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564062592:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826762.975346:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564062720:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826762.975381:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826762.975393:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826762/real 1787826762] req@ffff976889cf8f00 x1874670564062720/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826762.975404:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564062720/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826762.975412:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826762.975415:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826762.975418:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564062720:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826762.975423:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564062848:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826762.975441:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826762.975463:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826762/real 1787826762] req@ffff976889cfa940 x1874670564062848/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826762.975475:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564062848/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826762.975483:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826762.975486:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826762.975489:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564062848:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826762.975494:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564062976:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826762.975522:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826762.975534:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826762/real 1787826762] req@ffff976889cf9e00 x1874670564062976/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826762.975559:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564062976/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826762.975567:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826762.975578:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826762.975585:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564062976:192.168.203.139@tcp:8:kworker.0 00010000:00010000:3.0:1787826763.231034:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826763.998933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826764.254988:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826765.022954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826765.278956:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826766.047003:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826766.302939:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826767.070931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826767.328005:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826768.094945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826768.094977:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826768.094984:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826768.094988:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826768.094997:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1707 00000100:00080000:1.0:1787826768.095007:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826768.095013:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826768.095061:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564063104/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826768.095086:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826768.095090:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826768.095091:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826768.095095:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1707 00000100:00080000:1.0:1787826768.095100:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826768.095103:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826768.095113:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826768.095125:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564063232/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826768.095135:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787826768.095136:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564063104:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787826768.095140:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826768.095144:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826768.095149:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1707 00000100:00080000:1.0:1787826768.095153:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826768.095157:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826768.095163:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826768.095174:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564063360/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826768.095184:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826768.095186:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826768.095188:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826768.095192:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1707 00000100:00080000:1.0:1787826768.095195:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787826768.095206:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787826768.095211:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826768.095221:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826768.095234:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd680 x1874670564063488/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826768.095243:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826768.095247:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787826768.095249:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826768/real 1787826768] req@ffff976889efe580 x1874670564063104/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826768.095250:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826768.095257:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1707 00000100:00080000:1.0:1787826768.095264:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826768.095270:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826768.095275:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826768.095277:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564063104/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826768.095289:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564063616/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826768.095301:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826768.095309:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826768.095314:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564063104:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826768.095336:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564063232:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826768.095358:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826768.095370:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826768/real 1787826768] req@ffff976889eff480 x1874670564063232/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826768.095382:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564063232/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826768.095389:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826768.095392:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826768.095395:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564063232:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826768.095401:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564063360:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826768.095417:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826768.095426:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826768/real 1787826768] req@ffff976889efe940 x1874670564063360/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826768.095437:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564063360/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826768.095444:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826768.095447:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826768.095450:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564063360:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826768.095456:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564063488:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826768.095472:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826768.095481:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826768/real 1787826768] req@ffff976889efd680 x1874670564063488/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826768.095492:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd680 x1874670564063488/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826768.095499:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826768.095502:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826768.095505:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564063488:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826768.095510:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564063616:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826768.095527:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826768.095537:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826768/real 1787826768] req@ffff976889eff0c0 x1874670564063616/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826768.095547:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564063616/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826768.095568:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826768.095571:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826768.095575:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564063616:192.168.203.139@tcp:8:kworker.0 00010000:00010000:1.0:1787826768.351032:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826769.118937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826769.374966:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826770.142914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826770.399010:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826771.166948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826771.422970:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826772.190946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826772.446955:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826773.214955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826773.214971:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826773.214981:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826773.214986:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826773.214996:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1713 00000100:00080000:1.0:1787826773.215008:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826773.215016:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826773.215052:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564063744/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826773.215081:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826773.215084:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826773.215086:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826773.215090:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1713 00000100:00080000:1.0:1787826773.215094:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826773.215098:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826773.215108:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826773.215121:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564063744:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826773.215123:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564063872/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826773.215136:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826773.215140:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826773.215142:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826773.215146:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1713 00000100:00080000:1.0:1787826773.215151:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826773.215155:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826773.215160:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826773.215173:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564064000/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826773.215182:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787826773.215187:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787826773.215188:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826773.215192:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826773.215197:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1713 00000100:00080000:1.0:1787826773.215201:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826773.215205:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826773.215210:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826773.215224:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564064128/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826773.215233:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826773/real 1787826773] req@ffff976889efc3c0 x1874670564063744/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826773.215235:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826773.215243:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826773.215247:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826773.215256:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1713 00000100:00080000:1.0:1787826773.215261:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787826773.215264:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564063744/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826773.215267:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826773.215276:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787826773.215279:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826773.215284:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826773.215287:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564063744:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826773.215292:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd680 x1874670564064256/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826773.215308:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564063872:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826773.215333:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826773.215345:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826773/real 1787826773] req@ffff976889efed00 x1874670564063872/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826773.215358:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564063872/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826773.215365:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826773.215368:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826773.215371:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564063872:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826773.215377:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564064000:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826773.215397:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826773.215408:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826773/real 1787826773] req@ffff976889effc00 x1874670564064000/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826773.215419:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564064000/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826773.215428:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826773.215430:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826773.215432:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564064000:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826773.215436:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564064128:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826773.215453:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826773.215463:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826773/real 1787826773] req@ffff976889eff0c0 x1874670564064128/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826773.215474:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564064128/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826773.215482:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826773.215485:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826773.215488:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564064128:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826773.215494:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564064256:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826773.215509:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826773.215517:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826773/real 1787826773] req@ffff976889efd680 x1874670564064256/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826773.215523:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd680 x1874670564064256/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826773.215528:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826773.215530:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826773.215532:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564064256:192.168.203.139@tcp:8:kworker.0 00010000:00010000:2.0:1787826773.471086:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826774.238903:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826774.494984:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826775.262962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826775.518984:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826776.286935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826776.542979:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826777.310946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826777.567018:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787826778.335037:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826778.335048:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826778.335052:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826778.335060:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1718 00000100:00080000:1.0:1787826778.335070:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826778.335078:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826778.335125:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564064384/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826778.335156:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826778.335159:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826778.335161:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826778.335165:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1718 00000100:00080000:1.0:1787826778.335169:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826778.335173:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826778.335182:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826778.335193:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564064512/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826778.335202:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826778.335205:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826778.335207:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826778.335211:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1718 00000100:00080000:1.0:1787826778.335214:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826778.335217:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826778.335222:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826778.335232:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564064640/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826778.335247:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826778.335250:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826778.335252:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826778.335257:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1718 00000100:00080000:1.0:1787826778.335260:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826778.335264:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826778.335268:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826778.335279:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564064768/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787826778.335282:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826778.335292:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826778.335295:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826778.335298:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826778.335301:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1718 00000100:00080000:1.0:1787826778.335305:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826778.335320:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826778.335325:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826778.335332:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564064384:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826778.335338:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd2c0 x1874670564064896/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826778.335423:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564064512:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826778.335447:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564064640:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826778.335464:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564064768:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826778.335486:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564064896:192.168.203.139@tcp:8:kworker.0 00010000:00010000:3.0:1787826778.590956:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826779.358959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826779.615012:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826780.382943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826780.638979:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826781.406930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:2.0:1787826781.407248:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:2.0:1787826781.407309:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787826781.407330:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787826781.407379:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787826781.407390:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787826781.407414:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787826781.407424:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:0.0:1787826781.407478:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826778/real 1787826781] req@ffff976889efd2c0 x1874670564064896/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826781.407507:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd2c0 x1874670564064896/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826781.407519:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826781.407524:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826781.407529:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564064896:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826781.407538:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826778/real 1787826781] req@ffff976889efe940 x1874670564064384/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826781.407553:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564064384/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826781.407561:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826781.407565:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826781.407569:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564064384:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826781.407577:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826778/real 1787826781] req@ffff976889eff480 x1874670564064512/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826781.407588:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564064512/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826781.407595:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826781.407598:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826781.407601:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564064512:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826781.407607:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826778/real 1787826781] req@ffff976889efe580 x1874670564064640/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826781.407617:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564064640/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826781.407623:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826781.407627:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826781.407629:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564064640:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826781.407635:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826778/real 1787826781] req@ffff976889efcb40 x1874670564064768/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826781.407645:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564064768/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826781.407652:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826781.407656:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826781.407659:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564064768:192.168.203.139@tcp:8:kworker.0 00010000:00010000:3.0:1787826781.662988:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826782.430941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826782.687039:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826783.454964:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787826783.454972:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787826783.454975:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826783.454980:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826783.454991:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1723 00000100:00080000:3.0:1787826783.455002:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826783.455009:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787826783.455046:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564065024/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826783.455068:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826783.455070:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826783.455072:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826783.455074:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1723 00000100:00080000:3.0:1787826783.455076:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826783.455078:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826783.455087:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826783.455096:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564065152/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826783.455103:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826783.455106:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826783.455107:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826783.455110:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1723 00000100:00080000:3.0:1787826783.455113:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787826783.455114:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564065024:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787826783.455116:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826783.455123:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826783.455134:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564065280/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826783.455143:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826783.455146:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826783.455148:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826783.455152:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1723 00000100:00080000:3.0:1787826783.455156:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826783.455160:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826783.455165:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787826783.455175:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564065408/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826783.455184:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826783.455187:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826783.455188:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826783.455192:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1723 00000100:00080000:3.0:1787826783.455195:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787826783.455198:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787826783.455202:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826783.455209:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826783.455220:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564065536/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826783.455252:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826783/real 1787826783] req@ffff97688cb5c780 x1874670564065024/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826783.455271:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564065024/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826783.455283:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826783.455287:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826783.455292:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564065024:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826783.455320:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564065152:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826783.455344:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826783.455356:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826783/real 1787826783] req@ffff97688cb5cb40 x1874670564065152/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826783.455368:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564065152/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826783.455377:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826783.455380:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826783.455383:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564065152:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826783.455390:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564065280:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826783.455410:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826783.455422:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826783/real 1787826783] req@ffff97688cb5c000 x1874670564065280/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826783.455434:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564065280/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826783.456118:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826783.456124:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826783.456128:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564065280:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826783.456136:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564065408:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826783.456172:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826783.456188:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826783/real 1787826783] req@ffff97688cb5d680 x1874670564065408/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826783.456201:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564065408/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826783.456209:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826783.456213:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826783.456217:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564065408:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826783.456223:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564065536:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826783.456242:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826783.456253:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826783/real 1787826783] req@ffff97688cb5f0c0 x1874670564065536/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826783.456264:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564065536/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826783.456273:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826783.456276:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826783.456279:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564065536:192.168.203.139@tcp:8:kworker.0 00010000:00010000:1.0:1787826783.710995:0:49796:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826784.478943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826784.735491:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826785.503860:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826785.759207:0:29023:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 16/16 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1787826785.759268:0:49796:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 16 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1787826785.759276:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976888fb8400/0x8e7c99ee62ab0792 lrc: 2/0,0 mode: CR/CR res: [0x240000404:0xd:0x0].0x0 bits 0xa/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b80c0b expref: -99 pid: 53670 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787826785.759307:0:49796:0:(vvp_object.c:117:vvp_conf_set()) [0x240000404:0xd:0x0]: losing layout lock 00010000:00010000:1.0:1787826785.759335:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff9768c2169400/0x8e7c99ee62ab070d lrc: 2/0,0 mode: PR/PR res: [0x240000404:0xd:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x22153cd396b80abb expref: -99 pid: 53665 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826785.759360:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925de00/0x8e7c99ee62ab042e lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b80279 expref: -99 pid: 52441 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787826785.759371:0:49796:0:(vvp_object.c:117:vvp_conf_set()) [0x240000404:0x9:0x0]: losing layout lock 00010000:00010000:1.0:1787826785.759393:0:49796:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976880031a00/0x8e7c99ee62ab0435 lrc: 2/0,0 mode: CR/CR res: [0x240000404:0x9:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b8028e expref: -99 pid: 52442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000002:00100000:1.0:1787826785.759454:0:49796:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9768b01630c0 x1874670563702656/t4294967656(4294967656) o101->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 1096/1184 e 0 to 0 dl 1787826383 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lseek_test.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1787826785.759476:0:49796:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9768877a4800: modify RPC slot 2 is allocated opc 35, max 7 00000100:00100000:1.0:1787826785.759481:0:49796:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889efda40 x1874670564065792/t0(0) o35->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/2192 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00080000:1.0:1787826785.759501:0:49796:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff976889efda40 x1874670564065792/t0(0) o35->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/2192 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1787826785.759521:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787826785.759587:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826786.527616:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826786.782989:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826786.783245:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826787.551190:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787826787.807005:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787826787.807037:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826788.575798:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787826788.575883:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826788.575889:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787826788.575893:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826788.575903:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1728 00000100:00080000:2.0:1787826788.575917:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826788.575925:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826788.575962:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564065920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826788.575998:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826788.576002:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826788.576004:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826788.576008:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1728 00000100:00080000:2.0:1787826788.576012:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826788.576016:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826788.576026:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826788.576042:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9a40 x1874670564066048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826788.576052:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826788.576055:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787826788.576057:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826788.576064:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1728 00000100:00080000:2.0:1787826788.576067:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787826788.576080:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564065920:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787826788.576081:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826788.576103:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826788.576124:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564066176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826788.576140:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826788.576144:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826788.576146:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826788.576151:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1728 00000100:00080000:2.0:1787826788.576158:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826788.576162:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826788.576168:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787826788.576180:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564066304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826788.576191:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787826788.576196:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787826788.576201:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826788.576206:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826788.576213:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1728 00000100:00080000:2.0:1787826788.576220:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826788.576224:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826788.576230:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787826788.576243:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564066432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826788.576276:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826788/real 1787826788] req@ffff976889cfb0c0 x1874670564065920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826788.576310:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564065920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826788.576323:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826788.576329:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826788.576335:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564065920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826788.576344:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564066048:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826788.576389:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826788.576406:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826788/real 1787826788] req@ffff976889cf9a40 x1874670564066048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826788.576418:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9a40 x1874670564066048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826788.576426:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826788.576429:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826788.576432:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564066048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826788.576460:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564066176:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826788.576477:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826788.576485:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826788/real 1787826788] req@ffff976889cfb480 x1874670564066176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826788.576492:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564066176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826788.576502:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826788.576505:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826788.576507:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564066176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826788.576511:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564066304:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826788.576522:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826788.576529:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826788/real 1787826788] req@ffff976889cfa580 x1874670564066304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826788.576537:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564066304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826788.576542:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826788.576543:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826788.576545:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564066304:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787826788.576549:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564066432:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826788.576559:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826788.576570:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826788/real 1787826788] req@ffff976889cfa1c0 x1874670564066432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826788.576594:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564066432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826788.576603:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826788.576607:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826788.576611:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564066432:192.168.203.139@tcp:8:kworker.0 00010000:00010000:3.0:1787826788.830981:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826788.831427:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826789.598963:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787826789.854981:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787826789.855060:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826790.622990:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787826790.878933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826790.879002:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826791.646903:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787826791.902989:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787826791.903147:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826792.670997:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787826792.926933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826792.926960:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826793.694968:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787826793.694987:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787826793.694988:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826793.694995:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826793.695006:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1733 00000100:00080000:0.0:1787826793.695019:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826793.695028:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787826793.695070:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef840 x1874670564066560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826793.695095:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826793.695098:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826793.695100:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826793.695104:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1733 00000100:00080000:0.0:1787826793.695107:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826793.695112:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826793.695122:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826793.695134:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564066688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826793.695134:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564066560:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787826793.695144:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826793.695147:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826793.695149:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826793.695151:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1733 00000100:00080000:0.0:1787826793.695154:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826793.695157:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826793.695161:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826793.695178:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564066816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826793.695189:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826793.695192:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787826793.695192:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787826793.695194:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826793.695197:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1733 00000100:00080000:0.0:1787826793.695201:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826793.695204:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826793.695207:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787826793.695217:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564066944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826793.695225:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787826793.695227:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826793/real 1787826793] req@ffff976882aef840 x1874670564066560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826793.695228:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826793.695229:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826793.695233:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1733 00000100:00080000:0.0:1787826793.695236:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826793.695239:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826793.695243:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826793.695252:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564067072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826793.695252:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef840 x1874670564066560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826793.695262:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826793.695266:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826793.695270:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564066560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826793.695293:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564066688:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826793.695311:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826793.695321:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826793/real 1787826793] req@ffff976882aed680 x1874670564066688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826793.695337:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564066688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826793.695344:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826793.695347:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826793.695352:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564066688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826793.695358:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564066816:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826793.695377:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826793.695388:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826793/real 1787826793] req@ffff976882aec3c0 x1874670564066816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826793.695399:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564066816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826793.695407:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826793.695410:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826793.695414:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564066816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826793.695420:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564066944:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826793.695437:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826793.695446:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826793/real 1787826793] req@ffff976882aeed00 x1874670564066944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826793.695457:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564066944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826793.695465:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826793.695468:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826793.695471:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564066944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787826793.695477:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564067072:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826793.695494:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826793.695505:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826793/real 1787826793] req@ffff976882aee580 x1874670564067072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826793.695515:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564067072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826793.695522:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826793.695525:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826793.695528:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564067072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787826793.950932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787826793.951023:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826794.718995:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787826794.974941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826794.975418:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826795.743021:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787826795.998923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787826795.999000:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826796.766945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787826797.022922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826797.022950:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826797.791094:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787826798.046943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787826798.046982:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826798.814947:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826798.814959:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826798.814963:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787826798.814971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787826798.814973:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1738 00000100:00080000:0.0:1787826798.814985:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826798.814992:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787826798.815032:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564067200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826798.815059:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826798.815062:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826798.815064:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826798.815068:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1738 00000100:00080000:0.0:1787826798.815072:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826798.815076:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826798.815085:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826798.815096:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564067328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826798.815100:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564067200:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787826798.815106:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826798.815109:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826798.815110:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826798.815113:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1738 00000100:00080000:0.0:1787826798.815116:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826798.815118:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826798.815121:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826798.815128:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564067456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826798.815134:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826798.815137:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826798.815138:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826798.815140:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1738 00000100:00080000:0.0:1787826798.815148:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826798.815151:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:1.0:1787826798.815154:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:0.0:1787826798.815155:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787826798.815164:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564067584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826798.815170:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826798.815172:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826798.815173:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826798.815176:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1738 00000100:00080000:0.0:1787826798.815179:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826798.815181:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826798.815184:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787826798.815186:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826798/real 1787826798] req@ffff976882aee940 x1874670564067200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826798.815192:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564067712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826798.815216:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564067200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826798.815236:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826798.815240:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826798.815244:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564067200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826798.815265:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564067328:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826798.815296:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826798.815309:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826798/real 1787826798] req@ffff976882aef0c0 x1874670564067328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826798.815320:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564067328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826798.815328:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826798.815331:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826798.815335:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564067328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826798.815340:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564067456:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826798.815358:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826798.815367:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826798/real 1787826798] req@ffff976882aee580 x1874670564067456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826798.815378:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564067456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826798.815385:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826798.815388:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826798.815391:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564067456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826798.815396:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564067584:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826798.815413:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826798.815423:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826798/real 1787826798] req@ffff976882aeed00 x1874670564067584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826798.815433:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564067584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826798.815441:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826798.815444:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826798.815447:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564067584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787826798.815452:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564067712:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826798.815467:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826798.815476:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826798/real 1787826798] req@ffff976882aec3c0 x1874670564067712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826798.815487:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564067712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826798.815495:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826798.815498:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826798.815500:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564067712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787826799.070927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826799.070951:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826799.838960:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787826800.094932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787826800.095004:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826800.862933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787826801.118948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826801.118998:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826801.886893:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787826802.142931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787826802.142997:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826802.910938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787826803.166912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826803.166932:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787826803.934924:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787826803.934934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787826803.934934:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787826803.934940:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826803.934951:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1743 00000100:00080000:2.0:1787826803.934962:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826803.934969:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826803.935003:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564067840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826803.935025:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826803.935028:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826803.935030:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826803.935034:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1743 00000100:00080000:2.0:1787826803.935037:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826803.935042:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826803.935051:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826803.935064:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564067968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826803.935067:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564067840:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787826803.935077:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826803.935081:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787826803.935083:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826803.935087:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1743 00000100:00080000:2.0:1787826803.935090:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826803.935094:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826803.935099:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826803.935110:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564068096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826803.935120:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787826803.935125:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787826803.935125:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826803.935128:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826803.935133:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1743 00000100:00080000:2.0:1787826803.935137:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826803.935141:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826803.935146:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787826803.935156:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564068224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826803.935169:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826803.935173:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787826803.935176:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826803/real 1787826803] req@ffff976889cfad00 x1874670564067840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826803.935176:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826803.935183:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1743 00000100:00080000:2.0:1787826803.935188:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826803.935194:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826803.935200:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787826803.935203:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564067840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826803.935213:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564068352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826803.935218:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826803.935227:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826803.935232:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564067840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826803.935253:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564067968:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826803.935273:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826803.935285:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826803/real 1787826803] req@ffff976889cf92c0 x1874670564067968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826803.935293:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564067968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826803.935297:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826803.935299:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826803.935301:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564067968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826803.935304:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564068096:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826803.935316:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826803.935323:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826803/real 1787826803] req@ffff976889cfa1c0 x1874670564068096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826803.935333:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564068096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826803.935341:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826803.935344:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826803.935347:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564068096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826803.935353:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564068224:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826803.935379:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826803.935395:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826803/real 1787826803] req@ffff976889cfa580 x1874670564068224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826803.935407:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564068224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826803.935415:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826803.935418:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826803.935421:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564068224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787826803.935427:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564068352:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826803.935443:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826803.935451:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826803/real 1787826803] req@ffff976889cfb480 x1874670564068352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826803.935462:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564068352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826803.935469:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826803.935472:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826803.935475:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564068352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787826804.190915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787826804.191000:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826804.958949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787826805.214924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826805.214956:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826805.982951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787826806.238916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787826806.239003:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826807.006945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787826807.262934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826807.262965:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826808.030974:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787826808.286946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787826808.287112:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826809.054943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787826809.054946:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826809.054955:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826809.054959:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826809.054968:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1748 00000100:00080000:3.0:1787826809.054978:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826809.054986:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787826809.055021:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564068480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826809.055052:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826809.055056:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826809.055057:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826809.055061:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1748 00000100:00080000:3.0:1787826809.055064:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826809.055068:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826809.055078:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826809.055091:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564068608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826809.055100:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826809.055104:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826809.055106:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787826809.055110:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564068480:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787826809.055112:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1748 00000100:00080000:3.0:1787826809.055117:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826809.055122:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826809.055126:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826809.055137:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564068736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826809.055146:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826809.055149:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826809.055151:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826809.055155:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1748 00000100:00080000:3.0:1787826809.055160:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826809.055164:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826809.055169:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1787826809.055177:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826809.055182:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564068864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826809.055192:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826809.055195:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826809.055197:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826809.055201:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1748 00000100:00080000:3.0:1787826809.055204:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826809.055207:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826809.055213:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787826809.055221:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826809/real 1787826809] req@ffff97688cb5de00 x1874670564068480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826809.055225:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564068992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826809.055250:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564068480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826809.055263:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826809.055267:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826809.055271:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564068480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826809.055295:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564068608:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826809.055316:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826809.055328:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826809/real 1787826809] req@ffff97688cb5f0c0 x1874670564068608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826809.055340:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564068608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826809.055348:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826809.055351:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826809.055354:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564068608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826809.055360:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564068736:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826809.055377:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826809.055387:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826809/real 1787826809] req@ffff97688cb5d680 x1874670564068736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826809.055399:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564068736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826809.055408:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826809.055411:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826809.055414:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564068736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826809.055419:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564068864:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826809.055450:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826809.055461:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826809/real 1787826809] req@ffff97688cb5c000 x1874670564068864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826809.055471:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564068864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826809.055478:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826809.055481:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826809.055484:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564068864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787826809.055490:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564068992:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826809.055527:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826809.055538:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826809/real 1787826809] req@ffff97688cb5cb40 x1874670564068992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826809.055549:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564068992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826809.055563:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826809.055566:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826809.055569:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564068992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787826809.310936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826809.310967:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826810.079569:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:2.0:1787826810.335063:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826810.335098:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826811.102935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826811.358943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826811.358975:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826812.126965:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826812.382939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787826812.383007:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826813.150945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826813.406939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826813.406967:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826814.174936:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787826814.174938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787826814.174947:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826814.174951:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826814.174960:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1754 00000100:00080000:3.0:1787826814.174970:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826814.174978:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787826814.175014:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564069120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826814.175041:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826814.175045:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826814.175046:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826814.175051:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1754 00000100:00080000:3.0:1787826814.175054:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826814.175058:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826814.175069:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826814.175082:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564069248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826814.175091:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826814.175094:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826814.175096:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826814.175100:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1754 00000100:00080000:3.0:1787826814.175104:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826814.175107:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826814.175112:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826814.175120:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564069120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787826814.175130:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564069376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826814.175143:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826814.175147:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826814.175148:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826814.175154:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1754 00000100:00080000:3.0:1787826814.175158:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826814.175162:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826814.175168:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787826814.175182:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564069504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826814.175191:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826814.175195:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826814.175197:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826814.175201:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1754 00000100:00080000:3.0:1787826814.175205:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826814.175208:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:1.0:1787826814.175213:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:3.0:1787826814.175217:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826814.175234:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564069632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826814.175266:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826814/real 1787826814] req@ffff97688cb5c780 x1874670564069120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826814.175281:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564069120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826814.175290:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826814.175293:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826814.175296:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564069120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826814.175301:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564069248:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826814.175318:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826814.175326:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826814/real 1787826814] req@ffff97688cb5e1c0 x1874670564069248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826814.175333:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564069248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826814.175337:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826814.175339:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826814.175342:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564069248:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826814.175369:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564069376:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826814.175379:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826814.175389:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826814/real 1787826814] req@ffff97688cb5c3c0 x1874670564069376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826814.175399:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564069376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826814.175407:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826814.175410:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826814.175413:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564069376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826814.175420:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564069504:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826814.175439:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826814.175447:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826814/real 1787826814] req@ffff97688cb5f840 x1874670564069504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826814.175454:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564069504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826814.175459:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826814.175461:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826814.175463:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564069504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787826814.175467:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564069632:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826814.175495:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826814.175508:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826814/real 1787826814] req@ffff97688cb5cf00 x1874670564069632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826814.175519:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564069632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826814.175526:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826814.175530:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826814.175533:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564069632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826814.430944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787826814.431013:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826815.198937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826815.454936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826815.454966:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826816.222950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826816.478926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787826816.479003:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826817.246931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826817.502901:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826817.502934:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826818.270922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826818.526944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787826818.527058:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826819.294934:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787826819.294941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787826819.294944:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826819.294950:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826819.294959:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1759 00000100:00080000:3.0:1787826819.294970:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826819.294977:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787826819.295012:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564069760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826819.295044:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826819.295048:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826819.295049:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826819.295054:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1759 00000100:00080000:3.0:1787826819.295057:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826819.295061:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826819.295072:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826819.295087:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564069888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826819.295097:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826819.295101:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826819.295103:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826819.295107:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1759 00000100:00080000:3.0:1787826819.295110:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826819.295114:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826819.295119:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826819.295137:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564069760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787826819.295137:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564070016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826819.295161:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826819.295165:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826819.295167:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826819.295171:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1759 00000100:00080000:3.0:1787826819.295176:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826819.295180:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826819.295186:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787826819.295200:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564070144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826819.295210:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826819.295213:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826819.295215:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826819.295218:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1759 00000100:00080000:3.0:1787826819.295223:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826819.295226:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826819.295231:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826819.295243:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564070272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787826819.295253:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826819.295312:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826819/real 1787826819] req@ffff97688cb5d2c0 x1874670564069760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826819.295333:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564069760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826819.295345:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826819.295350:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826819.295354:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564069760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826819.295363:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564069888:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826819.295387:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826819.295407:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826819/real 1787826819] req@ffff97688cb5cf00 x1874670564069888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826819.295417:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564069888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826819.295431:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826819.295434:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826819.295437:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564069888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826819.295480:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564070016:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826819.295498:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826819.295507:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826819/real 1787826819] req@ffff97688cb5f840 x1874670564070016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826819.295519:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564070016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826819.295527:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826819.295530:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826819.295533:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564070016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826819.295539:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564070144:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826819.295557:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826819.295564:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826819/real 1787826819] req@ffff97688cb5c3c0 x1874670564070144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826819.295571:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564070144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826819.295576:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826819.295578:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826819.295581:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564070144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787826819.295587:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564070272:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826819.295604:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826819.295615:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826819/real 1787826819] req@ffff97688cb5e1c0 x1874670564070272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826819.295626:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564070272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826819.295631:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826819.295633:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826819.295635:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564070272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826819.550935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826819.551142:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826820.318936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826820.574934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826820.574971:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826821.342951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826821.598938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826821.599099:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826822.366966:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826822.622973:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826822.623009:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826823.390945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826823.646945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826823.646955:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826824.414939:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826824.414948:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826824.414953:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826824.414962:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1764 00000100:00080000:3.0:1787826824.414981:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826824.414993:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787826824.415043:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564070400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826824.415079:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826824.415082:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826824.415084:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826824.415088:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1764 00000100:00080000:3.0:1787826824.415092:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826824.415096:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826824.415113:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826824.415127:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564070400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787826824.415132:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564070528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826824.415145:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826824.415150:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826824.415151:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826824.415155:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1764 00000100:00080000:3.0:1787826824.415159:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826824.415163:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826824.415177:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787826824.415188:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826824.415195:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564070656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826824.415206:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826824.415209:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826824.415211:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826824.415216:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1764 00000100:00080000:3.0:1787826824.415221:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826824.415225:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826824.415230:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826824/real 1787826824] req@ffff97688cb5c780 x1874670564070400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787826824.415230:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787826824.415246:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564070784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826824.415252:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564070400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826824.415269:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826824.415272:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826824.415274:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826824.415278:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826824.415279:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:3.0:1787826824.415288:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1764 00000100:00100000:1.0:1787826824.415289:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564070400:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787826824.415295:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826824.415302:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826824.415307:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826824.415318:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564070912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826824.415322:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564070528:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826824.415360:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826824.415375:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826824/real 1787826824] req@ffff97688cb5cb40 x1874670564070528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826824.415387:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564070528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826824.415396:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826824.415399:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826824.415402:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564070528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826824.415408:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564070656:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826824.415441:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826824.415455:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826824/real 1787826824] req@ffff97688cb5c000 x1874670564070656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826824.415467:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564070656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826824.415475:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826824.415479:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826824.415484:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564070656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826824.415484:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787826824.415492:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564070784:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826824.415515:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826824.415525:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826824/real 1787826824] req@ffff97688cb5d680 x1874670564070784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826824.415538:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564070784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826824.415547:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826824.415551:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826824.415554:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564070784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787826824.415570:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564070912:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826824.415588:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826824.415599:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826824/real 1787826824] req@ffff97688cb5f0c0 x1874670564070912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826824.415610:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564070912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826824.415617:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826824.415621:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826824.415624:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564070912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826824.670930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826824.670957:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826825.438906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826825.694924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826825.694953:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826826.462948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826826.718895:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826826.718960:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826827.486946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826827.742941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826827.743021:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826828.510883:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826828.766907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826828.766943:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826829.534926:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826829.534936:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826829.534941:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826829.534951:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1769 00000100:00080000:3.0:1787826829.534961:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826829.534968:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787826829.535003:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564071040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826829.535032:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826829.535035:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826829.535037:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826829.535041:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1769 00000100:00080000:3.0:1787826829.535045:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826829.535048:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826829.535058:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826829.535071:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564071168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826829.535081:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826829.535083:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826829.535085:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826829.535088:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1769 00000100:00080000:3.0:1787826829.535090:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826829.535093:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826829.535096:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826829.535102:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564071296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826829.535107:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826829.535109:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826829.535111:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826829.535114:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1769 00000100:00080000:3.0:1787826829.535115:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826829.535118:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826829.535121:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787826829.535125:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564071424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826829.535133:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826829.535136:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787826829.535139:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564071040:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787826829.535139:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826829.535145:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1769 00000100:00080000:3.0:1787826829.535153:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826829.535157:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826829.535160:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826829.535164:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564071552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787826829.535241:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826829.535273:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787826829.535315:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826829/real 1787826829] req@ffff97688cb5de00 x1874670564071040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826829.535336:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564071040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826829.535347:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826829.535353:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826829.535357:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564071040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826829.535385:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564071168:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826829.535410:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826829.535424:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826829/real 1787826829] req@ffff97688cb5f0c0 x1874670564071168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826829.535436:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564071168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826829.535443:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826829.535446:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826829.535448:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564071168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826829.535455:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564071296:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826829.535472:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826829.535482:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826829/real 1787826829] req@ffff97688cb5d680 x1874670564071296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826829.535493:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564071296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826829.535501:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826829.535504:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826829.535507:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564071296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826829.535544:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564071424:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826829.535577:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826829.535589:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826829/real 1787826829] req@ffff97688cb5c000 x1874670564071424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826829.535600:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564071424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826829.535608:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826829.535611:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826829.535614:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564071424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787826829.535620:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564071552:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826829.535654:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826829.535665:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826829/real 1787826829] req@ffff97688cb5cb40 x1874670564071552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826829.535677:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564071552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826829.535685:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826829.535688:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826829.535691:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564071552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826829.791035:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826829.791064:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826830.558920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826830.814929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826830.815040:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826831.583195:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826831.839330:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826831.839869:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826832.606943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826832.862955:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826832.863021:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826833.630930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826833.886938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826833.886979:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826834.654932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787826834.654938:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826834.654945:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826834.654949:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826834.654959:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1774 00000100:00080000:3.0:1787826834.654971:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826834.654977:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787826834.655016:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564071680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826834.655038:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826834.655041:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826834.655043:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826834.655047:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1774 00000100:00080000:3.0:1787826834.655051:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826834.655055:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826834.655065:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826834.655073:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564071680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787826834.655079:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564071808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826834.655089:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826834.655093:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826834.655094:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826834.655098:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1774 00000100:00080000:3.0:1787826834.655101:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826834.655104:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826834.655108:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826834.655117:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564071936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826834.655126:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826834.655130:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826834.655133:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787826834.655135:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787826834.655140:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1774 00000100:00080000:3.0:1787826834.655145:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826834.655148:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826834.655153:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787826834.655161:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564072064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826834.655167:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826834.655170:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826834.655171:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826834.655175:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1774 00000100:00100000:1.0:1787826834.655177:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826834/real 1787826834] req@ffff97688cb5c780 x1874670564071680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826834.655177:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826834.655182:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826834.655189:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787826834.655203:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564071680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826834.655205:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564072192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826834.655237:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826834.655242:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826834.655246:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564071680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826834.655267:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564071808:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826834.655288:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826834.655301:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826834/real 1787826834] req@ffff97688cb5e1c0 x1874670564071808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826834.655311:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564071808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826834.655318:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826834.655321:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826834.655323:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564071808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826834.655328:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564071936:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826834.655366:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826834.655377:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826834/real 1787826834] req@ffff97688cb5c3c0 x1874670564071936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826834.655387:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564071936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826834.655394:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826834.655397:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826834.655399:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564071936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826834.655405:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564072064:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826834.655420:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826834.655428:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826834/real 1787826834] req@ffff97688cb5f840 x1874670564072064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826834.655437:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564072064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826834.655443:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826834.655446:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826834.655448:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564072064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787826834.655453:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564072192:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826834.655465:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826834.655473:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826834/real 1787826834] req@ffff97688cb5cf00 x1874670564072192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826834.655481:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564072192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826834.655488:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826834.655490:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826834.655493:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564072192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826834.910904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826834.910994:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826835.678938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826835.934965:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826835.935015:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826836.702944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826836.958906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826836.958973:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826837.727021:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826837.982932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826837.983011:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826838.750934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826839.006925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826839.006959:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787826839.774937:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787826839.774939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826839.774946:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826839.774950:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826839.774959:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1779 00000100:00080000:1.0:1787826839.774968:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826839.774976:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826839.775011:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564072320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826839.775035:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826839.775038:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826839.775040:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826839.775045:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1779 00000100:00080000:1.0:1787826839.775049:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826839.775052:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787826839.775059:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564072320:192.168.203.139@tcp:250:kworker.0 00000008:00100000:1.0:1787826839.775061:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826839.775074:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564072448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826839.775081:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826839.775082:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826839.775084:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826839.775086:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1779 00000100:00080000:1.0:1787826839.775088:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826839.775090:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826839.775092:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826839.775101:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564072576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826839.775111:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826839.775113:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826839.775114:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826839.775116:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1779 00000100:00080000:1.0:1787826839.775119:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787826839.775122:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787826839.775125:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826839.775130:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826839.775137:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564072704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826839.775142:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826839.775144:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826839.775145:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826839.775147:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1779 00000100:00080000:1.0:1787826839.775149:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826839.775150:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826839.775153:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787826839.775165:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564072832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826839.775166:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826839/real 1787826839] req@ffff976889efe1c0 x1874670564072320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826839.775198:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564072320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826839.775207:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826839.775212:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826839.775214:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564072320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826839.775237:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564072448:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826839.775254:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826839.775262:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826839/real 1787826839] req@ffff976889efc780 x1874670564072448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826839.775269:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564072448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826839.775274:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826839.775276:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826839.775278:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564072448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826839.775281:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564072576:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826839.775289:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826839.775294:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826839/real 1787826839] req@ffff976889efcb40 x1874670564072576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826839.775301:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564072576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826839.775320:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826839.775322:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826839.775324:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564072576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826839.775328:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564072704:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826839.775344:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826839.775354:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826839/real 1787826839] req@ffff976889efe580 x1874670564072704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826839.775365:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564072704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826839.775373:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826839.775376:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826839.775379:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564072704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826839.775384:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564072832:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826839.775401:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826839.775411:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826839/real 1787826839] req@ffff976889eff480 x1874670564072832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826839.775419:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564072832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826839.775423:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826839.775425:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826839.775427:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564072832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826840.030936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826840.030971:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826840.798953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826841.055446:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826841.056371:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826841.823047:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826842.078993:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826842.079002:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826842.846958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826843.103034:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826843.103092:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826843.870942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826844.126946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826844.126978:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826844.894938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826844.895081:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826844.895087:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826844.895091:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826844.895100:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1784 00000100:00080000:1.0:1787826844.895111:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826844.895117:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826844.895151:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564072960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826844.895175:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826844.895178:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826844.895180:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826844.895183:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1784 00000100:00080000:1.0:1787826844.895187:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826844.895191:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826844.895200:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826844.895211:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564072960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826844.895213:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd2c0 x1874670564073088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826844.895226:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826844.895230:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826844.895231:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826844.895235:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1784 00000100:00080000:1.0:1787826844.895239:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826844.895242:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826844.895247:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826844.895259:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd680 x1874670564073216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826844.895281:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826844.895285:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826844.895287:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826844.895291:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1784 00000100:00080000:1.0:1787826844.895294:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826844.895298:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826844.895303:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826844.895313:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564073344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826844.895322:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826844.895325:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826844.895327:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826844.895330:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1784 00000100:00080000:1.0:1787826844.895333:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826844.895337:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826844.895341:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787826844.895350:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564073472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826844.895365:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564073088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826844.895393:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564073216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826844.895411:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564073344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826844.895429:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564073472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826845.150938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826845.151008:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826845.918990:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826846.175037:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826846.175323:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826846.942943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826847.198988:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826847.199085:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826847.966939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:3.0:1787826847.967066:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:3.0:1787826847.967100:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787826847.967114:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826847.967149:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826847.967159:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826847.967180:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:0.0:1787826847.967193:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826844/real 1787826847] req@ffff976889efd2c0 x1874670564073088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787826847.967195:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:0.0:1787826847.967219:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd2c0 x1874670564073088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826847.967229:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826847.967234:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826847.967237:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564073088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826847.967244:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826844/real 1787826847] req@ffff976889efd680 x1874670564073216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826847.967311:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd680 x1874670564073216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826847.967318:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826847.967322:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826847.967325:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564073216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826847.967331:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826844/real 1787826847] req@ffff976889eff0c0 x1874670564073344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826847.967342:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564073344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826847.967348:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826847.967352:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826847.967355:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564073344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826847.967361:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826844/real 1787826847] req@ffff976889effc00 x1874670564073472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826847.967376:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564073472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826847.967383:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826847.967386:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826847.967389:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564073472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826847.967395:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826844/real 1787826847] req@ffff976889efe940 x1874670564072960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826847.967406:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564072960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826847.967413:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826847.967416:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826847.967419:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564072960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826848.222957:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826848.223036:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826848.991059:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826849.246895:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826849.246980:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826850.014911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826850.014911:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826850.014922:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826850.014926:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826850.014936:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1789 00000100:00080000:1.0:1787826850.014947:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826850.014954:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826850.014987:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564073600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826850.015016:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826850.015019:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826850.015022:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826850.015026:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1789 00000100:00080000:1.0:1787826850.015029:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826850.015034:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826850.015043:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826850.015053:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564073600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826850.015058:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564073728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826850.015069:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826850.015072:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826850.015074:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826850.015077:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1789 00000100:00080000:1.0:1787826850.015081:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826850.015085:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826850.015091:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826850.015103:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564073856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826850.015112:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787826850.015116:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787826850.015117:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826850.015121:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826850.015127:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1789 00000100:00080000:1.0:1787826850.015132:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826850.015136:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826850.015141:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826850.015152:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564073984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826850.015162:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787826850.015166:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826850/real 1787826850] req@ffff976889efed00 x1874670564073600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826850.015166:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826850.015169:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826850.015177:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1789 00000100:00080000:1.0:1787826850.015182:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826850.015186:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826850.015190:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826850.015192:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564073600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826850.015203:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564074112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826850.015216:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826850.015222:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826850.015227:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564073600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826850.015262:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564073728:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826850.015286:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826850.015299:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826850/real 1787826850] req@ffff976889efc3c0 x1874670564073728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826850.015310:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564073728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826850.015318:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826850.015321:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826850.015325:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564073728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826850.015331:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564073856:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826850.015349:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826850.015361:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826850/real 1787826850] req@ffff976889efe940 x1874670564073856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826850.015372:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564073856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826850.015380:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826850.015383:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826850.015387:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564073856:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826850.015392:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564073984:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826850.015411:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826850.015422:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826850/real 1787826850] req@ffff976889effc00 x1874670564073984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826850.015432:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564073984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826850.015440:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826850.015443:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826850.015446:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564073984:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826850.015452:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564074112:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826850.015470:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826850.015481:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826850/real 1787826850] req@ffff976889eff0c0 x1874670564074112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826850.015491:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564074112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826850.015498:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826850.015501:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826850.015503:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564074112:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826850.270928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787826850.270969:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826851.038924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826851.294935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826851.294966:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826852.062903:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826852.318925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787826852.318976:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826853.086956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826853.343006:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826853.344084:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826854.112231:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826854.366915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787826854.367491:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826855.134991:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826855.135000:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1787826855.135002:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787826855.135004:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826855.135015:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1795 00000100:00080000:3.0:1787826855.135025:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826855.135033:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787826855.135069:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564074240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826855.135102:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826855.135106:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826855.135108:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826855.135113:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1795 00000100:00080000:3.0:1787826855.135117:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826855.135121:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826855.135131:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826855.135132:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564074240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787826855.135150:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564074368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826855.135169:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826855.135172:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826855.135175:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826855.135178:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1795 00000100:00080000:3.0:1787826855.135182:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826855.135185:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826855.135189:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787826855.135206:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826855.135215:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564074496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826855.135225:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826855.135228:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826855.135230:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826855.135235:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1795 00000100:00080000:3.0:1787826855.135239:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826855.135247:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826855.135253:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787826855.135257:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826855/real 1787826855] req@ffff97688cb5d2c0 x1874670564074240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826855.135266:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564074624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826855.135278:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564074240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826855.135285:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826855.135288:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826855.135290:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826855.135295:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826855.135297:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:3.0:1787826855.135302:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1795 00000100:00100000:0.0:1787826855.135307:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564074240:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787826855.135307:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826855.135314:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826855.135319:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826855.135330:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564074752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826855.135336:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564074368:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826855.135364:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826855.135375:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826855/real 1787826855] req@ffff97688cb5cf00 x1874670564074368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826855.135382:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564074368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826855.135387:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826855.135389:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826855.135391:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564074368:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826855.135394:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564074496:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826855.135404:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826855.135410:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826855/real 1787826855] req@ffff97688cb5f840 x1874670564074496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826855.135416:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564074496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826855.135421:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826855.135422:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826855.135424:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564074496:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826855.135427:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564074624:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826855.135436:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826855.135442:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826855/real 1787826855] req@ffff97688cb5c3c0 x1874670564074624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826855.135448:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564074624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826855.135452:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826855.135454:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826855.135455:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564074624:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826855.135467:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564074752:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826855.135475:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826855.135481:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826855/real 1787826855] req@ffff97688cb5e1c0 x1874670564074752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826855.135487:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564074752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826855.135491:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826855.135493:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826855.135494:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564074752:192.168.203.139@tcp:8:kworker.0 00010000:00010000:3.0:1787826855.390955:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826855.391052:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826856.158933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826856.414888:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787826856.415008:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826857.182936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826857.438936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826857.438965:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826858.206920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826858.462949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787826858.463043:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826859.230978:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826859.486964:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826859.487010:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826860.254953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787826860.255009:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826860.255015:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826860.255019:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826860.255031:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1800 00000100:00080000:0.0:1787826860.255042:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826860.255051:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787826860.255086:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564074880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826860.255109:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826860.255112:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826860.255114:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826860.255116:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1800 00000100:00080000:0.0:1787826860.255120:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826860.255126:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826860.255136:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826860.255145:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef840 x1874670564075008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826860.255159:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787826860.255160:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564074880:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787826860.255183:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826860.255185:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826860.255190:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1800 00000100:00080000:0.0:1787826860.255195:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826860.255198:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826860.255203:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826860.255220:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564075136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826860.255229:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826860.255232:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826860.255234:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826860.255240:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1800 00000100:00080000:0.0:1787826860.255248:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826860.255251:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826860.255255:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787826860.255260:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826860.255268:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564075264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826860.255277:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826860.255280:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826860.255283:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826860.255286:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1800 00000100:00080000:0.0:1787826860.255289:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826860.255292:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826860.255296:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826860.255309:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564075392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787826860.255324:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826860/real 1787826860] req@ffff976882aed680 x1874670564074880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826860.255343:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564074880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826860.255352:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826860.255355:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826860.255358:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564074880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787826860.255470:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564075008:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787826860.255628:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826860.255640:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826860/real 1787826860] req@ffff976882aef840 x1874670564075008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826860.255654:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef840 x1874670564075008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826860.255715:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826860.255720:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826860.255724:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564075008:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787826860.255734:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564075136:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787826860.255757:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826860.255768:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826860/real 1787826860] req@ffff976882aec3c0 x1874670564075136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826860.255779:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564075136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826860.255786:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826860.255788:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826860.255790:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564075136:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787826860.255795:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564075264:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787826860.255810:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826860.255819:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826860/real 1787826860] req@ffff976882aeed00 x1874670564075264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826860.255861:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564075264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826860.255867:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826860.255870:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826860.255874:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564075264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787826860.255881:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564075392:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787826860.255904:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826860.255915:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826860/real 1787826860] req@ffff976882aee580 x1874670564075392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826860.255927:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564075392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826860.255933:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826860.255936:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787826860.255939:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564075392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826860.510970:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787826860.511038:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826861.278929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826861.534920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787826861.534961:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826862.302932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826862.558951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787826862.559098:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826863.326981:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826863.582953:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787826863.583024:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826864.350995:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826864.606932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787826864.607001:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826865.374936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787826865.374947:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826865.374954:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826865.374959:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826865.374969:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1805 00000100:00080000:3.0:1787826865.374979:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826865.374988:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787826865.375025:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564075520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826865.375054:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826865.375058:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826865.375060:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826865.375064:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1805 00000100:00080000:3.0:1787826865.375068:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826865.375072:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826865.375081:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826865.375093:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564075648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826865.375100:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564075520:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787826865.375106:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826865.375111:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826865.375112:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826865.375117:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1805 00000100:00080000:3.0:1787826865.375122:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826865.375139:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826865.375145:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826865.375157:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564075776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826865.375167:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826865.375169:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826865.375171:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826865.375174:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1805 00000800:00000100:0.0:1787826865.375180:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787826865.375181:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826865.375188:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826865.375194:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787826865.375205:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564075904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826865.375214:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826865.375217:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826865.375218:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826865.375222:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1805 00000100:00080000:3.0:1787826865.375226:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787826865.375229:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826865/real 1787826865] req@ffff97688cb5c780 x1874670564075520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826865.375233:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826865.375241:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826865.375255:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564075520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826865.375257:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564076032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826865.375283:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826865.375290:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826865.375295:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564075520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826865.375319:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564075648:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826865.375343:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826865.375356:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826865/real 1787826865] req@ffff97688cb5cb40 x1874670564075648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826865.375368:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564075648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826865.375375:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826865.375377:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826865.375380:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564075648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826865.375386:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564075776:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826865.375404:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826865.375415:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826865/real 1787826865] req@ffff97688cb5c000 x1874670564075776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826865.375425:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564075776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826865.375433:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826865.375435:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826865.375438:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564075776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826865.375444:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564075904:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826865.375462:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826865.375614:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826865/real 1787826865] req@ffff97688cb5d680 x1874670564075904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826865.375626:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564075904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826865.375634:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826865.375637:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826865.375640:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564075904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826865.375646:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564076032:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826865.375667:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826865.375678:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826865/real 1787826865] req@ffff97688cb5f0c0 x1874670564076032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826865.375688:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564076032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826865.375696:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826865.375699:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826865.375703:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564076032:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826865.630949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787826865.631091:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826866.398936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826866.655030:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787826866.655376:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826867.426263:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:0.0:1787826867.679040:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826867.679065:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826868.447314:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:0.0:1787826868.703017:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826868.703023:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826869.470915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826869.726936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787826869.727005:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826870.494945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826870.494974:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826870.494980:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826870.494984:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826870.494992:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1810 00000100:00080000:1.0:1787826870.495002:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826870.495010:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826870.495044:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd680 x1874670564076160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826870.495069:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826870.495074:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826870.495075:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826870.495079:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1810 00000100:00080000:1.0:1787826870.495082:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826870.495086:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826870.495109:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826870.495118:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564076160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826870.495124:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd2c0 x1874670564076288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826870.495135:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826870.495139:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826870.495141:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826870.495144:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1810 00000100:00080000:1.0:1787826870.495148:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826870.495152:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826870.495156:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826870.495168:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564076416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826870.495176:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826870.495179:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826870.495180:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826870.495184:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1810 00000100:00080000:1.0:1787826870.495188:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826870.495194:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:0.0:1787826870.495195:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787826870.495203:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826870.495216:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564076544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826870.495225:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826870.495228:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826870.495230:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826870.495234:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1810 00000100:00080000:1.0:1787826870.495238:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826870.495244:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826870.495248:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826870.495250:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826870/real 1787826870] req@ffff976889efd680 x1874670564076160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826870.495260:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564076672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826870.495280:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd680 x1874670564076160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826870.495292:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826870.495296:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826870.495300:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564076160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826870.495338:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564076288:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826870.495362:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826870.495376:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826870/real 1787826870] req@ffff976889efd2c0 x1874670564076288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826870.495388:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd2c0 x1874670564076288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826870.495396:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826870.495399:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826870.495402:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564076288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826870.495409:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564076416:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826870.495426:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826870.495437:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826870/real 1787826870] req@ffff976889eff480 x1874670564076416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826870.495448:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564076416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826870.495456:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826870.495459:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826870.495462:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564076416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826870.495467:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564076544:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826870.495486:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826870.495507:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826870/real 1787826870] req@ffff976889efe580 x1874670564076544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826870.495519:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564076544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826870.495527:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826870.495530:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826870.495533:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564076544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826870.495538:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564076672:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826870.495556:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826870.495568:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826870/real 1787826870] req@ffff976889efcb40 x1874670564076672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826870.495579:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564076672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826870.495586:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826870.495589:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826870.495592:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564076672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826870.750928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787826870.751014:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826871.518943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826871.774914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787826871.774964:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826872.543120:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826872.798938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787826872.799087:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826873.566980:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826873.822899:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787826873.822960:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826874.590943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826874.846937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787826874.847042:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826875.614938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787826875.615103:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826875.615109:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826875.615113:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826875.615122:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1815 00000100:00080000:3.0:1787826875.615136:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826875.615143:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787826875.615176:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564076800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826875.615200:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826875.615207:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826875.615209:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826875.615214:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1815 00000100:00080000:3.0:1787826875.615218:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826875.615222:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787826875.615233:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564076800:192.168.203.139@tcp:250:kworker.0 00000008:00100000:3.0:1787826875.615233:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826875.615251:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564076928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826875.615262:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826875.615266:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826875.615268:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826875.615272:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1815 00000100:00080000:3.0:1787826875.615276:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826875.615280:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826875.615285:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787826875.615299:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826875.615300:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564077056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826875.615312:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826875.615316:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826875.615319:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826875.615324:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1815 00000100:00080000:3.0:1787826875.615328:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826875.615332:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826875.615338:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787826875.615340:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826875/real 1787826875] req@ffff97688cb5de00 x1874670564076800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826875.615352:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564077184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826875.615357:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564076800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826875.615376:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826875.615377:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826875.615383:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:3.0:1787826875.615383:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826875.615387:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787826875.615389:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564076800:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787826875.615396:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1815 00000100:00080000:3.0:1787826875.615401:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826875.615405:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826875.615408:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826875.615416:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564076928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787826875.615421:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564077312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787826875.615445:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826875.615460:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826875/real 1787826875] req@ffff97688cb5f0c0 x1874670564076928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826875.615484:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564076928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826875.615492:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826875.615495:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826875.615498:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564076928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826875.615505:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564077056:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826875.615523:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826875.615532:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826875/real 1787826875] req@ffff97688cb5d680 x1874670564077056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826875.615544:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564077056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826875.615549:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826875.615551:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826875.615553:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564077056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826875.615557:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564077184:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826875.615568:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826875.615576:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826875/real 1787826875] req@ffff97688cb5c000 x1874670564077184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826875.615586:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564077184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826875.615593:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826875.615596:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826875.615599:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564077184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826875.615616:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564077312:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826875.615633:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826875.615641:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826875/real 1787826875] req@ffff97688cb5cb40 x1874670564077312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826875.615648:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564077312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826875.615656:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826875.615658:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826875.615660:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564077312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826875.870929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787826875.870988:0:42220:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826876.638936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826876.894936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826876.894975:0:29023:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 16/16 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1787826876.895044:0:42220:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 16 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787826876.895063:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff9768853fda00/0x8e7c99ee62ab09d0 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b81286 expref: -99 pid: 54858 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826876.895085:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1787826876.895092:0:42220:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x19:0x0]: losing layout lock 00010000:00010000:2.0:1787826876.895124:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff9768853fd000/0x8e7c99ee62ab09c9 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x19:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b8126a expref: -99 pid: 54857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787826876.895134:0:42220:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x19:0x0]: losing layout lock 00010000:00010000:2.0:1787826876.895142:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8600/0x8e7c99ee62ab0967 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x19:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x22153cd396b8116e expref: -99 pid: 54856 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826876.895161:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff9768853fde00/0x8e7c99ee62ab08d4 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x18:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x22153cd396b80fca expref: -99 pid: 54847 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826876.895177:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff9768853fce00/0x8e7c99ee62ab07ae lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x10:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x22153cd396b80cac expref: -99 pid: 54253 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826876.895187:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff9768c2168600/0x8e7c99ee62ab06ab lrc: 2/0,0 mode: PR/PR res: [0x200000bd0:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x22153cd396b80979 expref: -99 pid: 53656 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826876.895207:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff9768c2169c00/0x8e7c99ee62ab06c0 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x17:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x22153cd396b809aa expref: -99 pid: 53656 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826876.895221:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff9768c2169a00/0x8e7c99ee62ab06a4 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x17:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x22153cd396b8096b expref: -99 pid: 53656 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826876.895230:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8000/0x8e7c99ee62ab0443 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x13:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x22153cd396b802d4 expref: -99 pid: 53029 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787826876.895245:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8c00/0x8e7c99ee62ab01f7 lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x8:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x22153cd396b7fc13 expref: -99 pid: 52410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826876.895258:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925c000/0x8e7c99ee62ab01db lrc: 2/0,0 mode: PR/PR res: [0x200000405:0xe:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x22153cd396b7fb87 expref: -99 pid: 51827 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826876.895273:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb8e00/0x8e7c99ee62ab01aa lrc: 2/0,0 mode: CR/CR res: [0x200000405:0xf:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b7fad1 expref: -99 pid: 51813 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1787826876.895280:0:42220:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0xf:0x0]: losing layout lock 00010000:00010000:2.0:1787826876.895287:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff9768853fca00/0x8e7c99ee62ab01cd lrc: 2/0,0 mode: PR/PR res: [0x200000405:0xf:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x22153cd396b7fb5d expref: -99 pid: 51826 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787826876.895315:0:42220:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff9768ad0e0400/0x8e7c99ee62aafd82 lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x3:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x22153cd396b7f7cf expref: -99 pid: 47118 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000002:00100000:2.0:1787826876.895369:0:42220:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff976882aefc00 x1874670563573504/t4294969202(4294969202) o101->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 608/608 e 0 to 0 dl 1787826353 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787826876.895385:0:42220:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9768877a4800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787826876.895391:0:42220:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff976889cf9a40 x1874670564077568/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00080000:2.0:1787826876.895411:0:42220:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff976889cf9a40 x1874670564077568/t0(0) o35->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:23/10 lens 392/944 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1787826876.895419:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787826877.662943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826877.918920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787826877.918947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826877.918992:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826878.686964:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826878.942919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787826878.943024:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826878.943028:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826879.710938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826879.967068:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826879.967068:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826879.967080:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787826880.734946:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787826880.734947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826880.734957:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826880.734961:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826880.734969:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1820 00000100:00080000:1.0:1787826880.734978:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826880.734984:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826880.735016:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564077696/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826880.735045:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826880.735047:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826880.735049:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826880.735052:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1820 00000100:00080000:1.0:1787826880.735054:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826880.735057:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826880.735064:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826880.735074:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564077824/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826880.735080:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564077696:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787826880.735082:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826880.735089:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826880.735090:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826880.735094:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1820 00000100:00080000:1.0:1787826880.735095:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826880.735097:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826880.735100:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826880.735109:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564077952/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826880.735115:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826880.735117:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826880.735118:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826880.735121:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1820 00000100:00080000:1.0:1787826880.735140:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826880.735143:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826880.735145:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1787826880.735152:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826880.735154:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564078080/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826880.735164:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826880.735167:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826880.735169:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826880.735173:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1820 00000100:00080000:1.0:1787826880.735176:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826880.735180:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826880.735184:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826880.735198:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826880/real 1787826880] req@ffff976889efc780 x1874670564077696/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826880.735199:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564078208/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826880.735246:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564077696/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826880.735258:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826880.735261:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826880.735270:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564077696:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826880.735294:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564077824:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826880.735317:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826880.735332:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826880/real 1787826880] req@ffff976889efe1c0 x1874670564077824/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826880.735344:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564077824/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826880.735353:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826880.735356:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826880.735360:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564077824:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826880.735366:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564077952:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826880.735386:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826880.735398:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826880/real 1787826880] req@ffff976889efcb40 x1874670564077952/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826880.735410:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564077952/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826880.735417:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826880.735421:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826880.735424:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564077952:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826880.735429:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564078080:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826880.735449:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826880.735461:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826880/real 1787826880] req@ffff976889efe580 x1874670564078080/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826880.735472:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564078080/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826880.735480:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826880.735484:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826880.735487:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564078080:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826880.735493:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564078208:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826880.735512:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826880.735522:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826880/real 1787826880] req@ffff976889eff480 x1874670564078208/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826880.735533:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564078208/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826880.735540:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826880.735543:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826880.735546:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564078208:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826880.990942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826880.990960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826880.991043:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826881.759035:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826882.014933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826882.015123:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826882.015150:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787826882.782974:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826883.038967:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826883.038967:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826883.039034:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826883.806946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826884.062920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826884.062924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826884.062973:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826884.830975:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826885.086977:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826885.087068:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826885.087594:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826885.854954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826885.854954:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826885.854971:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826885.854976:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826885.854997:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1825 00000100:00080000:1.0:1787826885.855008:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826885.855015:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826885.855055:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd2c0 x1874670564078336/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826885.855082:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826885.855086:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826885.855087:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826885.855091:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1825 00000100:00080000:1.0:1787826885.855094:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826885.855099:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826885.855108:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826885.855122:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd680 x1874670564078464/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826885.855129:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564078336:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787826885.855136:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826885.855143:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826885.855145:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826885.855150:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1825 00000100:00080000:1.0:1787826885.855153:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826885.855156:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826885.855161:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826885.855175:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564078592/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826885.855184:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826885.855187:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826885.855189:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826885.855192:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1825 00000100:00080000:1.0:1787826885.855197:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787826885.855202:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787826885.855205:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826885.855212:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826885.855236:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564078720/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826885.855246:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826885.855250:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787826885.855252:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826885/real 1787826885] req@ffff976889efd2c0 x1874670564078336/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826885.855253:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826885.855264:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1825 00000100:00080000:1.0:1787826885.855270:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826885.855274:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787826885.855279:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd2c0 x1874670564078336/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787826885.855279:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787826885.855294:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564078848/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826885.855295:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826885.855305:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826885.855313:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564078336:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826885.855338:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564078464:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826885.855362:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826885.855377:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826885/real 1787826885] req@ffff976889efd680 x1874670564078464/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826885.855389:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd680 x1874670564078464/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826885.855397:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826885.855400:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826885.855403:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564078464:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826885.855409:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564078592:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826885.855427:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826885.855439:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826885/real 1787826885] req@ffff976889eff0c0 x1874670564078592/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826885.855456:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564078592/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826885.855464:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826885.855467:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826885.855470:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564078592:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826885.855475:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564078720:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826885.855493:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826885.855505:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826885/real 1787826885] req@ffff976889effc00 x1874670564078720/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826885.855516:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564078720/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826885.855523:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826885.855526:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826885.855528:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564078720:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826885.855540:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564078848:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826885.855558:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826885.855654:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826885/real 1787826885] req@ffff976889efe940 x1874670564078848/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826885.855667:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564078848/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826885.855680:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826885.855683:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826885.855685:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564078848:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826886.110917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826886.110917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826886.111003:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826886.878890:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826887.134933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826887.135117:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826887.136795:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826887.902949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826888.159058:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826888.161061:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826888.162028:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826888.926924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826889.182993:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826889.182997:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826889.183040:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787826889.950950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826890.206993:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826890.206994:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826890.207060:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826890.974920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826890.975024:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826890.975030:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826890.975034:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826890.975049:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1830 00000100:00080000:1.0:1787826890.975059:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826890.975066:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826890.975101:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564078976/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826890.975165:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564078976:192.168.203.139@tcp:250:kworker.0 00000800:00000100:0.0:1787826890.975223:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787826890.975254:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826890.975258:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826890.975260:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826890.975264:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1830 00000100:00080000:1.0:1787826890.975268:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826890.975272:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826890.975281:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826890.975302:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564079104/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826890.975370:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826890/real 1787826890] req@ffff976889efc3c0 x1874670564078976/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826890.975386:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564078976/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826890.975395:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826890.975399:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826890.975403:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564078976:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826890.975431:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564079104:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826890.975454:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826890.975478:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826890/real 1787826890] req@ffff976889efed00 x1874670564079104/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826890.975493:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564079104/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826890.975501:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826890.975504:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826890.975507:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564079104:192.168.203.139@tcp:38:kworker.0 00000100:00080000:1.0:1787826890.975554:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826890.975558:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826890.975560:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826890.975565:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1830 00000100:00080000:1.0:1787826890.975569:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826890.975572:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826890.975578:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826890.975591:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564079232/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826890.975600:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826890.975603:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826890.975605:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826890.975608:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1830 00000100:00080000:1.0:1787826890.975611:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826890.975615:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826890.975619:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826890.975627:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564079360/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826890.975635:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826890.975637:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826890.975639:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826890.975642:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1830 00000100:00080000:1.0:1787826890.975645:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826890.975648:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826890.975651:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787826890.975658:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564079488/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826890.975713:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564079232:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826890.975734:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826890.975746:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826890/real 1787826890] req@ffff976889efed00 x1874670564079232/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826890.975756:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564079232/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826890.975763:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826890.975766:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826890.975769:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564079232:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826890.975775:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564079360:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826890.975790:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826890.975799:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826890/real 1787826890] req@ffff976889efc3c0 x1874670564079360/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826890.975810:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564079360/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826890.975817:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826890.975820:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826890.975844:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564079360:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826890.975850:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564079488:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826890.975869:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826890.975882:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826890/real 1787826890] req@ffff976889efe940 x1874670564079488/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826890.975895:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564079488/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826890.975903:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826890.975907:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826890.975911:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564079488:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826891.230969:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826891.230969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826891.231082:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826891.998979:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826892.254958:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826892.254976:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826892.254990:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826893.023023:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826893.278915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826893.279026:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826893.279059:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826894.047019:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826894.302913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826894.302913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826894.303041:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826895.070938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826895.326893:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826895.326895:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826895.326961:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826896.094928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826896.094932:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826896.094940:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826896.094945:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826896.094954:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1835 00000100:00080000:1.0:1787826896.094964:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826896.094977:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826896.095018:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564079616/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826896.095038:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826896.095040:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826896.095041:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826896.095044:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1835 00000100:00080000:1.0:1787826896.095047:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826896.095050:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826896.095058:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826896.095067:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564079744/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826896.095078:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787826896.095079:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564079616:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787826896.095085:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826896.095089:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826896.095095:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1835 00000100:00080000:1.0:1787826896.095098:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826896.095101:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826896.095106:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826896.095119:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd680 x1874670564079872/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826896.095129:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826896.095131:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826896.095133:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826896.095138:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1835 00000100:00080000:1.0:1787826896.095141:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826896.095147:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:0.0:1787826896.095148:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787826896.095157:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826896.095170:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd2c0 x1874670564080000/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826896.095179:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826896.095182:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826896.095184:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826896.095187:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1835 00000100:00080000:1.0:1787826896.095190:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826896.095194:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826896.095199:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826896.095206:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826896/real 1787826896] req@ffff976889effc00 x1874670564079616/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826896.095213:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564080128/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826896.095233:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564079616/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826896.095243:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826896.095246:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826896.095249:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564079616:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826896.095269:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564079744:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826896.095294:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826896.095309:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826896/real 1787826896] req@ffff976889eff0c0 x1874670564079744/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826896.095322:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564079744/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826896.095331:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826896.095335:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826896.095338:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564079744:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826896.095345:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564079872:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826896.095365:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826896.095379:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826896/real 1787826896] req@ffff976889efd680 x1874670564079872/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826896.095396:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd680 x1874670564079872/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826896.095404:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826896.095407:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826896.095411:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564079872:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826896.095418:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564080000:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826896.095441:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826896.095454:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826896/real 1787826896] req@ffff976889efd2c0 x1874670564080000/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826896.095479:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd2c0 x1874670564080000/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826896.095488:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826896.095493:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826896.095496:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564080000:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826896.095503:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564080128:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826896.095524:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826896.095534:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826896/real 1787826896] req@ffff976889eff480 x1874670564080128/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826896.095542:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564080128/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826896.095551:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826896.095554:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826896.095556:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564080128:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826896.350919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826896.350922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826896.350993:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826897.118963:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826897.374949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826897.374971:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787826897.374984:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826898.142984:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826898.398997:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826898.399304:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826898.400227:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826899.166950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826899.422942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826899.422949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826899.422974:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826900.190967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826900.446958:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826900.446983:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826900.447071:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787826901.214936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826901.214936:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826901.214946:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826901.214950:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826901.214960:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1841 00000100:00080000:1.0:1787826901.214977:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826901.214988:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826901.215023:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564080256/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826901.215054:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826901.215057:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826901.215059:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826901.215063:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1841 00000100:00080000:1.0:1787826901.215067:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826901.215071:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826901.215081:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826901.215092:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564080256:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826901.215096:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564080384/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826901.215107:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826901.215111:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826901.215113:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826901.215117:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1841 00000100:00080000:1.0:1787826901.215120:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826901.215124:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826901.215129:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826901.215141:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564080512/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826901.215150:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826901.215154:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826901.215158:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787826901.215159:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787826901.215163:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1841 00000100:00080000:1.0:1787826901.215168:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826901.215172:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826901.215177:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826901.215189:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564080640/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826901.215199:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787826901.215202:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826901/real 1787826901] req@ffff976889efe580 x1874670564080256/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826901.215203:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826901.215207:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826901.215215:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1841 00000100:00080000:1.0:1787826901.215220:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787826901.215222:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564080256/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826901.215227:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826901.215235:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787826901.215238:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826901.215245:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826901.215252:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564080256:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826901.215252:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564080768/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826901.215282:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564080384:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826901.215305:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826901.215317:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826901/real 1787826901] req@ffff976889efcb40 x1874670564080384/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826901.215325:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564080384/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826901.215331:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826901.215339:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826901.215343:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564080384:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826901.215349:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564080512:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826901.215368:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826901.215378:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826901/real 1787826901] req@ffff976889efe1c0 x1874670564080512/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826901.215389:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564080512/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826901.215395:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826901.215397:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826901.215399:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564080512:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826901.215402:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564080640:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826901.215416:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826901.215423:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826901/real 1787826901] req@ffff976889efc780 x1874670564080640/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826901.215429:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564080640/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826901.215434:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826901.215436:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826901.215437:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564080640:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826901.215440:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564080768:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826901.215448:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826901.215454:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826901/real 1787826901] req@ffff976889eff480 x1874670564080768/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826901.215459:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564080768/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826901.215463:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826901.215465:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826901.215467:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564080768:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826901.470945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826901.470946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787826901.470983:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826902.238953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826902.494966:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787826902.495010:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826902.495026:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826903.263023:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826903.518944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826903.518944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826903.518974:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826904.286938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826904.542937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826904.542942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826904.542960:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826905.310947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787826905.566974:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826905.567074:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826905.569023:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00080000:1.0:1787826906.334919:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787826906.334925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826906.334927:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826906.334935:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826906.334945:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1846 00000100:00080000:1.0:1787826906.334956:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826906.334964:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826906.335002:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd2c0 x1874670564080896/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826906.335025:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826906.335029:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826906.335030:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826906.335034:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1846 00000100:00080000:1.0:1787826906.335037:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826906.335040:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826906.335047:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826906.335058:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd680 x1874670564081024/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826906.335066:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826906.335069:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787826906.335070:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564080896:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787826906.335071:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826906.335079:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1846 00000100:00080000:1.0:1787826906.335083:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826906.335086:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826906.335089:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826906.335095:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564081152/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826906.335101:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826906.335103:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826906.335104:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826906.335107:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1846 00000100:00080000:1.0:1787826906.335109:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826906.335111:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826906.335115:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826906.335121:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564081280/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826906.335126:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826906.335128:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826906.335129:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826906.335131:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1846 00000100:00080000:1.0:1787826906.335133:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787826906.335137:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787826906.335139:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826906.335143:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787826906.335150:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564081408/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826906.335176:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826906/real 1787826906] req@ffff976889efd2c0 x1874670564080896/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826906.335192:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd2c0 x1874670564080896/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826906.335202:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826906.335206:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826906.335210:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564080896:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826906.335230:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564081024:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826906.335260:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826906.335273:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826906/real 1787826906] req@ffff976889efd680 x1874670564081024/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826906.335285:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd680 x1874670564081024/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826906.335292:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826906.335295:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826906.335297:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564081024:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826906.335301:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564081152:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826906.335314:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826906.335324:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826906/real 1787826906] req@ffff976889eff0c0 x1874670564081152/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826906.335340:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564081152/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826906.335348:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826906.335351:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826906.335354:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564081152:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826906.335359:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564081280:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826906.335376:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826906.335383:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826906/real 1787826906] req@ffff976889effc00 x1874670564081280/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826906.335390:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564081280/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826906.335394:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826906.335396:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826906.335398:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564081280:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826906.335401:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564081408:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826906.335410:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826906.335423:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826906/real 1787826906] req@ffff976889efe940 x1874670564081408/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826906.335430:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564081408/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826906.335438:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826906.335440:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826906.335442:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564081408:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826906.590932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826906.590932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826906.590981:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826907.358968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826907.614957:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826907.614958:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787826907.614974:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826908.382960:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826908.638979:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826908.638986:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826908.639031:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826909.407007:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826909.662939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826909.662939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826909.662981:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826910.431048:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826910.686920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826910.686925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787826910.686955:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826911.454954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826911.454959:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826911.454969:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826911.454972:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826911.454982:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1851 00000100:00080000:1.0:1787826911.454992:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826911.454999:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826911.455031:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564081536/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826911.455050:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826911.455053:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826911.455054:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826911.455057:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1851 00000100:00080000:1.0:1787826911.455060:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826911.455064:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826911.455073:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826911.455081:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564081664/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826911.455089:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787826911.455092:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564081536:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787826911.455093:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826911.455099:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826911.455107:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1851 00000100:00080000:1.0:1787826911.455109:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826911.455111:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826911.455114:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826911.455122:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564081792/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826911.455127:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826911.455129:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826911.455130:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826911.455133:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1851 00000100:00080000:1.0:1787826911.455135:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826911.455137:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826911.455140:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826911.455147:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564081920/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826911.455156:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826911.455158:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826911.455159:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826911.455162:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1851 00000100:00080000:1.0:1787826911.455164:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826911.455166:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826911.455168:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787826911.455175:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564082048/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826911.455224:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564081664:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826911.455257:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564081792:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826911.455277:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564081920:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826911.455297:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564082048:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826911.710992:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826911.710993:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787826911.711031:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826912.478933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826912.734961:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826912.734984:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826912.735050:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787826913.502937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826913.758964:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826913.758964:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826913.759028:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826914.526931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:3.0:1787826914.527021:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:3.0:1787826914.527048:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787826914.527060:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826914.527087:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826914.527094:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826914.527100:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787826914.527107:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:0.0:1787826914.527117:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826911/real 1787826914] req@ffff976889efed00 x1874670564081664/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826914.527139:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564081664/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826914.527149:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826914.527154:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826914.527158:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564081664:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826914.527165:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826911/real 1787826914] req@ffff976889efe940 x1874670564081792/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826914.527177:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564081792/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826914.527184:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826914.527188:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826914.527191:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564081792:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826914.527198:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826911/real 1787826914] req@ffff976889effc00 x1874670564081920/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826914.527209:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564081920/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826914.527217:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826914.527220:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826914.527223:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564081920:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826914.527233:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826911/real 1787826914] req@ffff976889eff0c0 x1874670564082048/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826914.527243:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564082048/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826914.527251:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826914.527254:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826914.527257:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564082048:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826914.527264:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826911/real 1787826914] req@ffff976889efc3c0 x1874670564081536/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826914.527275:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564081536/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826914.527283:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826914.527286:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826914.527289:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564081536:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826914.782923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826914.782923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826914.782954:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826915.550991:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826915.806926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826915.806926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826915.806955:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787826916.574939:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826916.574948:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787826916.574952:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826916.574961:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1856 00000100:00100000:0.0:1787826916.574965:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787826916.574970:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826916.574978:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826916.575012:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564082176/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826916.575030:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826916.575033:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826916.575034:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826916.575037:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1856 00000100:00080000:2.0:1787826916.575039:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826916.575044:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826916.575053:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826916.575061:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564082176:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826916.575067:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564082304/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826916.575086:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826916.575089:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787826916.575091:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826916.575095:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1856 00000100:00080000:2.0:1787826916.575100:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826916.575104:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826916.575109:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787826916.575129:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787826916.575136:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564082432/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826916.575148:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826916.575151:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826916.575152:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826916.575155:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1856 00000100:00080000:2.0:1787826916.575159:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826916.575162:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826916.575166:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787826916.575170:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826916/real 1787826916] req@ffff976889cfb0c0 x1874670564082176/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826916.575187:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564082560/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826916.575199:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564082176/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826916.575199:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826916.575208:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826916.575212:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826916.575219:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787826916.575220:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1856 00000100:00080000:2.0:1787826916.575225:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826916.575231:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:0.0:1787826916.575233:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826916.575239:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564082176:192.168.203.139@tcp:250:kworker.0 00000008:00100000:2.0:1787826916.575251:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826916.575263:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564082304:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787826916.575267:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564082688/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787826916.575294:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826916.575309:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826916/real 1787826916] req@ffff976889cf9e00 x1874670564082304/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826916.575321:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564082304/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826916.575329:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826916.575332:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826916.575335:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564082304:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826916.575340:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564082432:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826916.575360:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826916.575371:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826916/real 1787826916] req@ffff976889cfa940 x1874670564082432/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826916.575397:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564082432/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826916.575411:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826916.575414:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826916.575418:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564082432:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826916.575424:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564082560:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826916.575444:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826916.575456:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826916/real 1787826916] req@ffff976889cf8f00 x1874670564082560/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826916.575467:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564082560/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826916.575474:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826916.575477:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826916.575480:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564082560:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826916.575494:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564082688:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826916.575513:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826916.575523:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826916/real 1787826916] req@ffff976889cfbc00 x1874670564082688/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826916.575533:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564082688/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826916.575541:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826916.575544:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826916.575548:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564082688:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826916.830920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826916.830942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787826916.830978:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826917.599010:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826917.854912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787826917.855005:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826917.855024:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787826918.622937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826918.878947:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787826918.878956:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826918.879025:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826919.646934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826919.902931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826919.902998:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826919.903047:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826920.670937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826920.926909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826920.926965:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826920.926988:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00080000:1.0:1787826921.694918:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826921.694926:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826921.694930:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787826921.694931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826921.694941:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1861 00000100:00080000:1.0:1787826921.694950:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826921.694957:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826921.694992:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd680 x1874670564082816/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826921.695013:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826921.695015:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826921.695016:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826921.695020:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1861 00000100:00080000:1.0:1787826921.695023:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826921.695026:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826921.695034:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826921.695044:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd2c0 x1874670564082944/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826921.695051:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564082816:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787826921.695053:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826921.695059:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826921.695062:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826921.695065:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1861 00000100:00080000:1.0:1787826921.695067:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826921.695069:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826921.695071:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826921.695079:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564083072/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826921.695085:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826921.695087:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826921.695088:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826921.695091:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1861 00000100:00080000:1.0:1787826921.695093:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826921.695095:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826921.695098:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826921.695104:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564083200/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826921.695110:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826921.695112:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826921.695113:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826921.695114:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1861 00000100:00080000:1.0:1787826921.695118:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787826921.695122:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787826921.695129:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826921.695134:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787826921.695146:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564083328/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826921.695164:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826921/real 1787826921] req@ffff976889efd680 x1874670564082816/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826921.695181:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd680 x1874670564082816/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826921.695191:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826921.695195:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826921.695199:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564082816:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826921.695220:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564082944:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826921.695238:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826921.695247:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826921/real 1787826921] req@ffff976889efd2c0 x1874670564082944/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826921.695254:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd2c0 x1874670564082944/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826921.695260:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826921.695262:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826921.695264:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564082944:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826921.695268:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564083072:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826921.695277:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826921.695283:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826921/real 1787826921] req@ffff976889eff480 x1874670564083072/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826921.695289:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564083072/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826921.695294:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826921.695295:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826921.695297:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564083072:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826921.695300:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564083200:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826921.695309:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826921.695314:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826921/real 1787826921] req@ffff976889efc780 x1874670564083200/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826921.695321:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564083200/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826921.695325:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826921.695326:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826921.695328:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564083200:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826921.695331:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564083328:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826921.695339:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826921.695344:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826921/real 1787826921] req@ffff976889efe1c0 x1874670564083328/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826921.695349:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564083328/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826921.695353:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826921.695355:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826921.695356:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564083328:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826921.950885:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826921.950891:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826921.950955:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826922.718927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826922.974937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826922.974955:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787826922.974967:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826923.742918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826923.998912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826923.998912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826923.998942:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826924.766979:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826925.022919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826925.022919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826925.022948:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826925.790909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826926.046930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826926.046967:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787826926.046978:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787826926.814952:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826926.814962:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787826926.814967:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826926.814977:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1866 00000100:00080000:2.0:1787826926.814989:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826926.814996:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826926.815040:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564083456/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826926.815064:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826926.815067:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826926.815069:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826926.815074:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1866 00000100:00080000:2.0:1787826926.815077:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826926.815082:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826926.815091:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826926.815117:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564083584/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826926.815127:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826926.815130:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787826926.815132:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826926.815136:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1866 00000100:00080000:2.0:1787826926.815140:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826926.815144:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826926.815149:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826926.815153:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564083456:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826926.815163:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564083712/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826926.815174:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826926.815178:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826926.815180:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826926.815183:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1866 00000100:00080000:2.0:1787826926.815187:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826926.815191:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826926.815196:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787826926.815208:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564083840/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826926.815217:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787826926.815222:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787826926.815223:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826926.815227:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826926.815232:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1866 00000100:00080000:2.0:1787826926.815236:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826926.815240:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826926.815245:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787826926.815257:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564083968/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826926.815272:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826926/real 1787826926] req@ffff976889cf8000 x1874670564083456/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826926.815290:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564083456/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826926.815299:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826926.815304:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826926.815308:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564083456:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826926.815314:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564083584:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826926.815334:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826926.815346:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826926/real 1787826926] req@ffff976889cfbc00 x1874670564083584/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826926.815357:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564083584/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826926.815364:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826926.815367:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826926.815370:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564083584:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826926.815391:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564083712:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826926.815408:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826926.815418:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826926/real 1787826926] req@ffff976889cf8f00 x1874670564083712/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826926.815428:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564083712/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826926.815435:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826926.815438:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826926.815441:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564083712:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826926.815446:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564083840:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826926.815464:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826926.815473:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826926/real 1787826926] req@ffff976889cfa940 x1874670564083840/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826926.815487:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564083840/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826926.815495:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826926.815499:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826926.815502:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564083840:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826926.815507:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564083968:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826926.815531:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826926.815542:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826926/real 1787826926] req@ffff976889cf9e00 x1874670564083968/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826926.815552:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564083968/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826926.815557:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826926.815559:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826926.815561:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564083968:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787826926.815620:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826927.070930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826927.070930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826927.070993:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826927.838977:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826928.094922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826928.094922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826928.094949:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826928.862937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826929.118938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826929.118998:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826929.119124:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787826929.886957:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826930.142956:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826930.143020:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826930.143076:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787826930.910970:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826931.166963:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826931.167075:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826931.168559:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00080000:2.0:1787826931.935544:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826931.935558:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787826931.935565:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826931.935578:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1871 00000100:00080000:2.0:1787826931.935593:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826931.935602:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826931.935649:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564084096/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826931.935691:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826931.935695:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826931.935697:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826931.935701:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1871 00000100:00080000:2.0:1787826931.935705:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826931.935709:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826931.935721:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826931.935733:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564084224/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826931.935744:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826931.935749:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787826931.935751:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826931.935756:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1871 00000100:00080000:2.0:1787826931.935759:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826931.935764:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826931.935769:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826931.935781:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564084352/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826931.935798:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826931.935801:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826931.935803:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826931.935809:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1871 00000100:00080000:2.0:1787826931.935812:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826931.935816:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826931.935821:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787826931.935869:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564084480/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826931.935879:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826931.935883:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826931.935885:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826931.935890:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1871 00000100:00080000:2.0:1787826931.935894:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826931.935898:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826931.935905:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826931.935914:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564084096:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826931.935923:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564084608/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787826931.935992:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826931.936029:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826931/real 1787826931] req@ffff976889cfb0c0 x1874670564084096/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826931.936047:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564084096/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826931.936060:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826931.936064:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826931.936070:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564084096:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826931.936078:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564084224:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826931.936098:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826931.936109:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826931/real 1787826931] req@ffff976889cfb480 x1874670564084224/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826931.936120:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564084224/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826931.936141:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826931.936144:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826931.936147:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564084224:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826931.936153:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564084352:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826931.936171:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826931.936182:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826931/real 1787826931] req@ffff976889cfa580 x1874670564084352/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826931.936200:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564084352/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826931.936207:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826931.936211:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826931.936215:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564084352:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787826931.936218:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826931.936232:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564084480:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826931.936251:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826931.936262:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826931/real 1787826931] req@ffff976889cfa1c0 x1874670564084480/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826931.936272:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564084480/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826931.936279:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826931.936283:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826931.936286:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564084480:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826931.936322:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564084608:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826931.936343:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826931.936359:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826931/real 1787826931] req@ffff976889cf92c0 x1874670564084608/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826931.936370:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564084608/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826931.936383:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826931.936386:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826931.936389:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564084608:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826932.190963:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787826932.190967:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826932.191084:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826932.958944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826933.214980:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826933.215018:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826933.215116:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787826933.982945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826934.238963:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826934.238973:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826934.239069:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826935.007012:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826935.262931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826935.262931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826935.263003:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826936.030974:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826936.286916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826936.286916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826936.287042:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787826937.054945:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787826937.054950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787826937.054954:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787826937.054958:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826937.054966:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1876 00000100:00080000:2.0:1787826937.054974:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826937.054980:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826937.055016:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564084736/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826937.055043:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826937.055046:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826937.055048:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826937.055052:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1876 00000100:00080000:2.0:1787826937.055055:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826937.055059:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826937.055069:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826937.055086:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564084736:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787826937.055095:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564084864/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826937.055111:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826937.055114:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787826937.055116:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826937.055120:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1876 00000100:00080000:2.0:1787826937.055124:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826937.055127:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826937.055132:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826937.055145:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564084992/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787826937.055153:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787826937.055156:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826937.055160:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826937.055161:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826937.055165:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1876 00000100:00080000:2.0:1787826937.055169:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826937.055173:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826937.055176:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787826937.055186:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564085120/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826937.055194:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826937.055197:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826937.055199:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826937.055203:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1876 00000100:00080000:2.0:1787826937.055207:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826937.055210:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826937.055216:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826937.055219:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826937/real 1787826937] req@ffff976889cfad00 x1874670564084736/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826937.055229:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564085248/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826937.055241:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564084736/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826937.055254:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826937.055268:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826937.055278:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564084736:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826937.055302:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564084864:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826937.055323:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826937.055336:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826937/real 1787826937] req@ffff976889cf92c0 x1874670564084864/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826937.055346:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564084864/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826937.055354:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826937.055357:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826937.055360:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564084864:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826937.055365:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564084992:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826937.055383:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826937.055393:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826937/real 1787826937] req@ffff976889cfa1c0 x1874670564084992/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826937.055404:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564084992/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826937.055412:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826937.055415:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826937.055418:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564084992:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826937.055423:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564085120:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826937.055441:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826937.055451:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826937/real 1787826937] req@ffff976889cfa580 x1874670564085120/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826937.055461:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564085120/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826937.055468:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826937.055471:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826937.055475:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564085120:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826937.055480:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564085248:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826937.055498:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826937.055509:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826937/real 1787826937] req@ffff976889cfb480 x1874670564085248/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826937.055518:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564085248/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826937.055525:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826937.055528:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826937.055530:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564085248:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826937.310891:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826937.310904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826937.310955:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826938.079314:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826938.334962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787826938.335104:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826938.335131:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787826939.102964:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826939.358978:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826939.358983:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787826939.359082:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826940.127059:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826940.382926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787826940.385744:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826940.385797:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787826941.150898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826941.406966:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826941.407053:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826941.407064:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787826942.174924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787826942.174942:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826942.174948:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826942.174954:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826942.174964:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1882 00000100:00080000:3.0:1787826942.174974:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826942.174981:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787826942.175020:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564085376/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826942.175040:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826942.175043:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826942.175044:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826942.175047:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1882 00000100:00080000:3.0:1787826942.175049:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826942.175052:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826942.175059:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826942.175068:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564085504/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826942.175075:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826942.175077:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826942.175078:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826942.175080:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1882 00000100:00080000:3.0:1787826942.175081:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826942.175083:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826942.175085:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826942.175091:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564085632/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826942.175098:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787826942.175099:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564085376:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787826942.175100:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826942.175104:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826942.175110:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1882 00000100:00080000:3.0:1787826942.175113:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826942.175115:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826942.175118:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787826942.175134:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564085760/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826942.175151:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826942.175157:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826942.175159:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826942.175163:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1882 00000100:00080000:3.0:1787826942.175168:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787826942.175169:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787826942.175175:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826942.175181:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826942.175192:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564085888/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826942.175214:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826942/real 1787826942] req@ffff97688cb5c780 x1874670564085376/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826942.175232:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564085376/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826942.175242:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826942.175247:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826942.175259:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564085376:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826942.175266:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564085504:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826942.175290:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826942.175314:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826942/real 1787826942] req@ffff97688cb5e1c0 x1874670564085504/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826942.175326:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564085504/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826942.175334:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826942.175337:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826942.175340:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564085504:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826942.175367:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564085632:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826942.175389:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826942.175400:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826942/real 1787826942] req@ffff97688cb5c3c0 x1874670564085632/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826942.175411:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564085632/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826942.175419:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826942.175422:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826942.175426:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564085632:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826942.175432:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564085760:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826942.175452:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826942.175464:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826942/real 1787826942] req@ffff97688cb5f840 x1874670564085760/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826942.175475:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564085760/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826942.175482:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826942.175486:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826942.175488:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564085760:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826942.175494:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564085888:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826942.175512:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826942.175522:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826942/real 1787826942] req@ffff97688cb5cf00 x1874670564085888/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826942.175533:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564085888/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826942.175540:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826942.175543:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826942.175545:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564085888:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826942.430945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787826942.430993:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787826942.431014:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826943.198952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826943.454939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787826943.455036:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826943.455146:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787826944.223656:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826944.478969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826944.478973:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787826944.479177:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826945.247020:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826945.502932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826945.502932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826945.503194:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826946.271095:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826946.526935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826946.526991:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826946.527033:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826947.294967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826947.295497:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826947.295502:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826947.295506:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826947.295515:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1887 00000100:00080000:1.0:1787826947.295526:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826947.295533:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826947.295583:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564086016/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826947.295623:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826947.295626:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826947.295628:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826947.295631:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1887 00000100:00080000:1.0:1787826947.295634:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826947.295637:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826947.295646:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826947.295655:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564086144/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826947.295665:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826947.295668:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826947.295669:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826947.295671:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1887 00000100:00080000:1.0:1787826947.295673:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826947.295675:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826947.295678:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826947.295686:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564086272/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826947.295694:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564086016:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787826947.295694:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826947.295700:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826947.295704:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826947.295707:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1887 00000100:00080000:1.0:1787826947.295709:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826947.295711:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826947.295713:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826947.295721:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564086400/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826947.295726:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826947.295728:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826947.295729:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826947.295731:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1887 00000100:00080000:1.0:1787826947.295734:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826947.295736:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826947.295739:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787826947.295745:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564086528/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787826947.296091:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826947.296134:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826947/real 1787826947] req@ffff976889efcb40 x1874670564086016/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826947.296151:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564086016/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826947.296160:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826947.296164:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826947.296167:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564086016:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826947.296174:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564086144:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826947.296194:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826947.296205:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826947/real 1787826947] req@ffff976889efe580 x1874670564086144/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826947.296215:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564086144/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826947.296223:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826947.296226:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826947.296229:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564086144:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826947.296249:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564086272:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826947.296266:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826947.296276:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826947/real 1787826947] req@ffff976889efe1c0 x1874670564086272/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826947.296286:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564086272/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826947.296293:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826947.296295:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826947.296298:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564086272:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826947.296303:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564086400:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826947.296318:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826947.296326:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826947/real 1787826947] req@ffff976889efc780 x1874670564086400/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826947.296336:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564086400/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826947.296343:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826947.296346:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826947.296349:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564086400:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826947.296354:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564086528:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826947.296369:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826947.296377:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826947/real 1787826947] req@ffff976889eff480 x1874670564086528/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826947.296387:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564086528/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826947.296399:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826947.296401:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826947.296404:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564086528:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826947.550936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826947.550986:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826947.551476:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826948.318929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826948.574936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826948.574956:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826948.575071:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826949.342944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826949.599029:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826949.599039:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826949.599703:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787826950.366946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826950.622931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826950.622931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826950.622995:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826951.390937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826951.646975:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826951.646985:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826951.647077:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787826952.414938:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826952.414949:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826952.414952:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826952.414963:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1892 00000100:00080000:1.0:1787826952.414974:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826952.414981:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826952.415015:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd2c0 x1874670564086656/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826952.415035:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826952.415037:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826952.415038:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826952.415041:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1892 00000100:00080000:1.0:1787826952.415045:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826952.415047:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826952.415055:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826952.415063:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd680 x1874670564086784/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826952.415070:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826952.415072:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826952.415073:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826952.415076:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1892 00000100:00080000:1.0:1787826952.415077:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826952.415079:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826952.415082:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826952.415091:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564086912/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826952.415099:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564086656:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787826952.415104:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826952.415108:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826952.415109:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826952.415114:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1892 00000100:00080000:1.0:1787826952.415118:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826952.415122:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826952.415127:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826952.415138:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564087040/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826952.415151:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826952.415154:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826952.415156:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826952.415159:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1892 00000800:00000100:0.0:1787826952.415176:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787826952.415216:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787826952.415257:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826952.415261:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826952.415266:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787826952.415276:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564087168/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826952.415295:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826952/real 1787826952] req@ffff976889efd2c0 x1874670564086656/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826952.415324:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd2c0 x1874670564086656/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826952.415334:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826952.415338:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826952.415342:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564086656:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826952.415350:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564086784:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826952.415373:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826952.415386:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826952/real 1787826952] req@ffff976889efd680 x1874670564086784/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826952.415399:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd680 x1874670564086784/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826952.415408:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826952.415411:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826952.415414:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564086784:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826952.415437:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564086912:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826952.415459:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826952.415471:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826952/real 1787826952] req@ffff976889efc3c0 x1874670564086912/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826952.415482:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564086912/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826952.415489:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826952.415493:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826952.415496:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564086912:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826952.415502:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564087040:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826952.415543:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826952.415556:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826952/real 1787826952] req@ffff976889eff0c0 x1874670564087040/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826952.415568:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564087040/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826952.415576:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826952.415579:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826952.415582:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564087040:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826952.415589:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564087168:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826952.415608:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826952.415619:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826952/real 1787826952] req@ffff976889effc00 x1874670564087168/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826952.415629:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564087168/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826952.415637:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826952.415640:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826952.415643:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564087168:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826952.670941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826952.670966:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826952.670998:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826953.438947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826953.694951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826953.694951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787826953.695041:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826954.462938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826954.719027:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787826954.719032:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826954.719071:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826955.486958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826955.743051:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787826955.743056:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826955.743151:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826956.510999:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826956.766940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787826956.766942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826956.767018:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787826957.534923:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826957.534932:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787826957.534937:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826957.534947:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1897 00000100:00080000:1.0:1787826957.534957:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826957.534966:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787826957.535003:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564087296/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826957.535034:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826957.535038:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787826957.535041:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826957.535045:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1897 00000100:00080000:1.0:1787826957.535048:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826957.535053:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826957.535063:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826957.535075:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787826957.535246:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564087424/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826957.535254:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826957.535257:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787826957.535258:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826957.535261:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1897 00000100:00080000:1.0:1787826957.535264:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826957.535266:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826957.535271:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826957.535280:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564087552/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826957.535285:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826957.535287:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787826957.535288:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826957.535291:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1897 00000100:00080000:1.0:1787826957.535292:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826957.535294:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826957.535296:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826957.535302:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564087680/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826957.535307:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826957.535309:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787826957.535310:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826957.535311:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1897 00000100:00080000:1.0:1787826957.535313:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826957.535315:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787826957.535317:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787826957.535321:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564087808/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826957.535479:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564087296:192.168.203.139@tcp:250:kworker.0 00000800:00000100:0.0:1787826957.535543:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826957.535574:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826957/real 1787826957] req@ffff976889efe940 x1874670564087296/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826957.535590:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564087296/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826957.535600:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826957.535604:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826957.535607:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564087296:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826957.535614:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564087424:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826957.535634:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826957.535647:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826957/real 1787826957] req@ffff976889efed00 x1874670564087424/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826957.535658:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564087424/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826957.535666:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826957.535669:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826957.535672:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564087424:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826957.535677:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564087552:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826957.535695:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826957.535706:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826957/real 1787826957] req@ffff976889effc00 x1874670564087552/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826957.535717:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564087552/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826957.535724:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826957.535727:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826957.535730:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564087552:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826957.535735:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564087680:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826957.535754:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826957.535765:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826957/real 1787826957] req@ffff976889eff0c0 x1874670564087680/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826957.535776:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564087680/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826957.535783:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826957.535786:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826957.535789:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564087680:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826957.535794:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564087808:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826957.535812:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826957.536039:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826957/real 1787826957] req@ffff976889efc3c0 x1874670564087808/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826957.536054:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564087808/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826957.536061:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826957.536065:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826957.536068:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564087808:192.168.203.139@tcp:8:kworker.0 00010000:00010000:1.0:1787826957.791088:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826957.791196:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787826957.791713:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787826958.559166:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826958.814963:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826958.814987:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787826958.815192:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787826959.584303:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787826959.840618:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787826959.840746:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826959.840799:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826960.606958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826960.863019:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787826960.863102:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826960.863530:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826961.630912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826961.886933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826961.886984:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826961.887565:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787826962.654892:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787826962.654960:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826962.654966:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826962.654970:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826962.654977:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1902 00000100:00080000:3.0:1787826962.654986:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826962.654991:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787826962.655020:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564087936/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826962.655043:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826962.655045:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826962.655047:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826962.655049:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1902 00000100:00080000:3.0:1787826962.655052:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826962.655054:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826962.655062:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826962.655070:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564088064/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826962.655077:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826962.655081:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826962.655082:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826962.655090:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1902 00000100:00080000:3.0:1787826962.655093:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826962.655097:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826962.655101:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826962.655111:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564088192/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826962.655119:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826962.655122:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826962.655124:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826962.655127:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1902 00000100:00080000:3.0:1787826962.655130:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826962.655133:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826962.655138:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787826962.655148:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564088320/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826962.655155:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826962.655158:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826962.655159:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826962.655162:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1902 00000100:00080000:3.0:1787826962.655165:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826962.655168:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826962.655172:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826962.655193:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564088448/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826962.655475:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564087936:192.168.203.139@tcp:250:kworker.0 00000800:00000100:0.0:1787826962.655532:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826962.655575:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826962/real 1787826962] req@ffff97688cb5d2c0 x1874670564087936/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826962.655594:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564087936/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826962.655603:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826962.655606:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826962.655609:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564087936:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826962.655615:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564088064:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826962.655632:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826962.655644:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826962/real 1787826962] req@ffff97688cb5cf00 x1874670564088064/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826962.655655:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564088064/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826962.655662:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826962.655665:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826962.655669:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564088064:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826962.655675:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564088192:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787826962.655693:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826962.655701:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826962/real 1787826962] req@ffff97688cb5f840 x1874670564088192/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826962.655709:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564088192/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826962.655718:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826962.655720:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826962.655722:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564088192:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826962.655725:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564088320:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826962.655736:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826962.655742:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826962/real 1787826962] req@ffff97688cb5c3c0 x1874670564088320/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826962.655748:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564088320/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826962.655754:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826962.655756:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826962.655759:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564088320:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826962.655764:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564088448:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787826962.655780:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787826962.655789:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826962/real 1787826962] req@ffff97688cb5e1c0 x1874670564088448/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826962.655799:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564088448/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826962.655806:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826962.655809:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826962.655813:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564088448:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787826962.911156:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826962.911409:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826962.911667:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787826963.678947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826963.935025:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826963.935079:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826963.935115:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787826964.703551:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787826964.958991:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826964.958999:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826964.959509:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787826965.726971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826965.982961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787826965.982962:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826965.983478:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826966.750934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826967.006988:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826967.007081:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826967.007089:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826967.774963:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826967.774978:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826967.774985:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787826967.774988:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787826967.775001:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1907 00000100:00080000:3.0:1787826967.775015:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826967.775024:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787826967.775078:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564088576/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826967.775112:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826967.775116:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826967.775118:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826967.775123:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1907 00000100:00080000:3.0:1787826967.775128:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826967.775134:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826967.775149:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826967.775157:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564088576:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787826967.775173:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564088704/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826967.775190:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826967.775194:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826967.775196:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826967.775201:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1907 00000100:00080000:3.0:1787826967.775207:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826967.775211:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826967.775217:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826967.775236:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564088832/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787826967.775238:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826967.775291:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826967/real 1787826967] req@ffff97688cb5c780 x1874670564088576/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826967.775317:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564088576/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826967.775344:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787826967.775350:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787826967.775352:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826967.775363:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564088576:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826967.775388:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564088704:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826967.775412:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826967.775445:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826967/real 1787826967] req@ffff97688cb5cb40 x1874670564088704/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826967.775462:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564088704/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826967.775470:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826967.775473:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826967.775476:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564088704:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826967.775482:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564088832:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826967.775502:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826967.775520:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826967/real 1787826967] req@ffff97688cb5c000 x1874670564088832/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826967.775528:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787826967.775535:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564088832/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826967.775535:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826967.775548:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1907 00000100:00080000:3.0:1787826967.775556:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787826967.775574:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826967.775579:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826967.775583:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564088832:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787826967.775608:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826967.775618:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787826967.775637:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564088960/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826967.775656:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826967.775660:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826967.775662:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826967.775666:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1907 00000100:00080000:3.0:1787826967.775671:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826967.775676:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826967.775682:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787826967.775683:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564088960:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787826967.775699:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564089088/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787826967.775718:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826967.775744:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826967/real 1787826967] req@ffff97688cb5d680 x1874670564088960/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826967.775754:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564088960/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826967.775763:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826967.775766:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826967.775769:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564088960:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787826967.775782:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564089088:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826967.775811:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826967.775854:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826967/real 1787826967] req@ffff97688cb5f0c0 x1874670564089088/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826967.775868:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564089088/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826967.775875:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826967.775877:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826967.775880:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564089088:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826968.031029:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826968.031118:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826968.031136:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787826968.799037:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826969.054920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826969.054944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826969.054970:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826969.822945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826970.078918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826970.078919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826970.078983:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826970.846962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826971.102881:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826971.102883:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826971.103094:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826971.870943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826972.126926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826972.126927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826972.127086:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826972.894945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787826972.894966:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826972.894975:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787826972.894980:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826972.894989:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1912 00000100:00080000:2.0:1787826972.894998:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826972.895005:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826972.895038:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564089216/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826972.895062:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826972.895066:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826972.895067:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826972.895071:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1912 00000100:00080000:2.0:1787826972.895074:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826972.895077:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826972.895087:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826972.895098:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564089344/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826972.895101:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564089216:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787826972.895115:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826972.895118:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787826972.895120:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826972.895124:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1912 00000100:00080000:2.0:1787826972.895127:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826972.895131:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826972.895135:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826972.895145:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564089472/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826972.895159:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826972.895162:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826972.895163:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826972.895168:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1912 00000800:00000100:1.0:1787826972.895170:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787826972.895173:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826972.895178:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826972.895184:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787826972.895196:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564089600/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826972.895205:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826972.895208:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826972.895212:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787826972.895213:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826972/real 1787826972] req@ffff976889cfb0c0 x1874670564089216/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787826972.895217:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1912 00000100:00080000:2.0:1787826972.895223:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826972.895230:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826972.895235:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787826972.895247:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564089216/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787826997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787826972.895248:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564089728/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787826972.895280:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826972.895284:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826972.895290:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564089216:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826972.895313:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564089344:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826972.895334:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826972.895346:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826972/real 1787826972] req@ffff976889cf9e00 x1874670564089344/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826972.895358:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564089344/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826972.895375:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826972.895387:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826972.895391:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564089344:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826972.895397:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564089472:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826972.895412:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826972.895420:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826972/real 1787826972] req@ffff976889cfa940 x1874670564089472/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826972.895427:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564089472/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787826997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826972.895432:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826972.895434:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826972.895436:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564089472:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826972.895439:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564089600:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826972.895448:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826972.895453:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826972/real 1787826972] req@ffff976889cf8f00 x1874670564089600/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826972.895459:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564089600/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826972.895464:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826972.895465:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826972.895468:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564089600:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787826972.895471:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564089728:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826972.895483:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826972.895492:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826972/real 1787826972] req@ffff976889cfbc00 x1874670564089728/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826972.895502:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564089728/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787826997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826972.895509:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826972.895512:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826972.895515:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564089728:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787826973.150961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826973.151420:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826973.151456:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787826973.919555:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826974.174980:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826974.174990:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826974.175225:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826974.943283:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826975.198937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826975.198937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826975.199166:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826975.966996:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826976.222964:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826976.222964:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826976.223073:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826976.990953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826977.246940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826977.246940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826977.247042:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787826978.014918:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826978.014929:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787826978.014933:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787826978.014935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787826978.014944:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1917 00000100:00080000:2.0:1787826978.014955:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826978.014962:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826978.014997:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564089856/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826978.015021:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826978.015024:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826978.015025:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826978.015029:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1917 00000100:00080000:2.0:1787826978.015033:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826978.015036:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826978.015046:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826978.015059:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564089984/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826978.015067:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826978.015071:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787826978.015072:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826978.015076:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1917 00000100:00080000:2.0:1787826978.015080:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826978.015083:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826978.015088:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826978.015096:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564090112/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826978.015101:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564089856:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787826978.015105:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826978.015109:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826978.015110:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826978.015114:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1917 00000100:00080000:2.0:1787826978.015116:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826978.015117:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826978.015120:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787826978.015129:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564090240/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826978.015134:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826978.015137:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826978.015138:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826978.015139:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1917 00000100:00080000:2.0:1787826978.015143:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826978.015144:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826978.015147:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787826978.015152:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564090368/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826978.015192:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564089984:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826978.015233:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564090112:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826978.015259:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564090240:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787826978.015280:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564090368:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826978.270968:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826978.271052:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826978.271065:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826979.039813:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826979.294929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826979.294929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826979.295040:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826980.063016:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826980.318930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826980.318930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826980.319003:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826981.087128:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:1.0:1787826981.087154:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:1.0:1787826981.087194:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787826981.087211:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787826981.087245:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787826981.087253:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787826981.087261:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787826981.087269:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:0.0:1787826981.087287:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826978/real 1787826981] req@ffff976889cf8f00 x1874670564090112/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826981.087309:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564090112/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826981.087319:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826981.087336:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826981.087340:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564090112:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826981.087348:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826978/real 1787826981] req@ffff976889cfa940 x1874670564090240/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826981.087360:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564090240/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826981.087368:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826981.087372:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826981.087376:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564090240:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826981.087381:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826978/real 1787826981] req@ffff976889cf9e00 x1874670564090368/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826981.087392:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564090368/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826981.087400:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826981.087403:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826981.087406:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564090368:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826981.087412:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826978/real 1787826981] req@ffff976889cf8000 x1874670564089856/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826981.087422:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564089856/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826981.087430:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826981.087433:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826981.087436:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564089856:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826981.087442:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826978/real 1787826981] req@ffff976889cfbc00 x1874670564089984/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826981.087452:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564089984/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826981.087460:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826981.087464:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787826981.087466:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564089984:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787826981.342933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826981.342933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826981.343059:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826982.111039:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826982.366933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787826982.367043:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826982.367296:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826983.134946:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826983.134957:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826983.134961:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787826983.134962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787826983.134972:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1923 00000100:00080000:0.0:1787826983.134981:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826983.134987:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787826983.135024:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564090496/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826983.135054:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826983.135056:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826983.135058:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826983.135061:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1923 00000100:00080000:0.0:1787826983.135065:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826983.135069:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826983.135078:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787826983.135098:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564090496:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787826983.135108:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564090624/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826983.135120:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826983.135123:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826983.135125:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826983.135129:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1923 00000100:00080000:0.0:1787826983.135133:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826983.135137:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826983.135143:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826983.135163:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564090752/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787826983.135163:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787826983.135175:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826983.135179:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826983.135181:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826983.135185:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1923 00000100:00080000:0.0:1787826983.135189:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826983.135193:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826983.135199:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787826983.135203:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826983/real 1787826983] req@ffff976882aef0c0 x1874670564090496/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826983.135214:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564090880/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826983.135223:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564090496/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826983.135225:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826983.135235:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826983.135238:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787826983.135243:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787826983.135245:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1923 00000100:00080000:1.0:1787826983.135250:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:0.0:1787826983.135252:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787826983.135257:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564090496:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787826983.135259:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787826983.135267:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826983.135277:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564091008/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826983.135288:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564090624:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826983.135329:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826983.135343:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826983/real 1787826983] req@ffff976882aee940 x1874670564090624/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826983.135360:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564090624/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826983.135367:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826983.135370:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826983.135373:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564090624:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826983.135379:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564090752:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826983.135397:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826983.135406:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826983/real 1787826983] req@ffff976882aee580 x1874670564090752/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826983.135416:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564090752/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826983.135422:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826983.135425:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826983.135428:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564090752:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826983.135432:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564090880:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826983.135456:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826983.135466:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826983/real 1787826983] req@ffff976882aeed00 x1874670564090880/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826983.135475:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564090880/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826983.135483:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826983.135485:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826983.135488:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564090880:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787826983.135501:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564091008:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826983.135518:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826983.135528:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826983/real 1787826983] req@ffff976882aec3c0 x1874670564091008/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826983.135538:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564091008/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826983.135544:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826983.135547:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826983.135550:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564091008:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826983.390930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826983.390930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826983.390994:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826984.159151:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826984.414910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826984.414910:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826984.415176:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826985.182952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826985.438929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826985.438929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826985.439028:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826986.206967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826986.462946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826986.463065:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826986.463851:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787826987.230962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826987.486932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826987.487001:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826987.487106:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00080000:3.0:1787826988.254922:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826988.254933:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1787826988.254934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787826988.254937:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826988.254945:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1928 00000100:00080000:3.0:1787826988.254954:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826988.254961:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787826988.254995:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564091136/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826988.255016:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826988.255018:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826988.255019:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826988.255022:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1928 00000100:00080000:3.0:1787826988.255025:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826988.255027:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826988.255034:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826988.255045:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564091264/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826988.255052:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564091136:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787826988.255054:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826988.255059:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826988.255060:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826988.255063:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1928 00000100:00080000:3.0:1787826988.255065:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826988.255066:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826988.255069:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826988.255077:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564091392/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826988.255082:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826988.255084:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826988.255085:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826988.255088:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1928 00000100:00080000:3.0:1787826988.255090:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826988.255092:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826988.255095:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787826988.255101:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564091520/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826988.255106:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826988.255109:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826988.255110:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826988.255115:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1928 00000800:00000100:1.0:1787826988.255116:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787826988.255118:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826988.255122:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826988.255125:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826988.255131:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564091648/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826988.255156:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826988/real 1787826988] req@ffff97688cb5de00 x1874670564091136/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826988.255172:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564091136/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826988.255180:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826988.255183:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826988.255185:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564091136:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826988.255204:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564091264:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826988.255220:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826988.255228:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826988/real 1787826988] req@ffff97688cb5f0c0 x1874670564091264/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826988.255235:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564091264/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826988.255240:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826988.255242:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826988.255244:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564091264:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826988.255247:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564091392:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826988.255255:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826988.255260:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826988/real 1787826988] req@ffff97688cb5d680 x1874670564091392/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826988.255267:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564091392/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826988.255284:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826988.255286:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826988.255288:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564091392:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826988.255292:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564091520:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826988.255307:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826988.255313:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826988/real 1787826988] req@ffff97688cb5c000 x1874670564091520/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826988.255320:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564091520/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826988.255325:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826988.255326:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826988.255328:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564091520:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787826988.255331:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564091648:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826988.255345:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826988.255354:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826988/real 1787826988] req@ffff97688cb5cb40 x1874670564091648/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826988.255364:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564091648/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826988.255381:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826988.255384:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826988.255387:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564091648:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826988.510889:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826988.510974:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826988.510982:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787826989.278971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826989.534962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826989.534988:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826989.535054:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787826990.302900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826990.558986:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826990.559039:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826990.559083:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826991.326968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826991.582944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826991.582951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826991.583039:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826992.350944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826992.606934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826992.606934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826992.607013:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826993.374946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787826993.376279:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826993.376285:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787826993.376288:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826993.376296:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1933 00000100:00080000:2.0:1787826993.376304:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826993.376311:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787826993.376348:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564091776/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826993.376376:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826993.376379:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787826993.376380:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826993.376383:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1933 00000100:00080000:2.0:1787826993.376386:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826993.376388:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826993.376397:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826993.376409:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564091904/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826993.376424:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826993.376427:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787826993.376429:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826993.376432:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1933 00000100:00080000:2.0:1787826993.376435:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826993.376439:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826993.376443:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787826993.376456:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564092032/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826993.376464:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826993.376467:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787826993.376469:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826993.376473:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1933 00000100:00080000:2.0:1787826993.376476:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826993.376480:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826993.376485:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787826993.376493:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564092160/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787826993.376501:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787826993.376504:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787826993.376505:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787826993.376508:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1933 00000100:00080000:2.0:1787826993.376512:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787826993.376515:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787826993.376519:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787826993.376528:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564092288/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826993.376611:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564091776:192.168.203.139@tcp:250:kworker.0 00000800:00000100:1.0:1787826993.376667:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826993.376707:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826993/real 1787826993] req@ffff976889cfb0c0 x1874670564091776/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826993.376722:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564091776/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826993.376732:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826993.376735:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826993.376739:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564091776:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826993.376745:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564091904:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826993.376766:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826993.376779:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826993/real 1787826993] req@ffff976889cfb480 x1874670564091904/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826993.376790:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564091904/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826993.376798:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826993.376801:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826993.376804:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564091904:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826993.376810:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564092032:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826993.376848:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826993.376860:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826993/real 1787826993] req@ffff976889cfa580 x1874670564092032/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826993.376871:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564092032/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826993.376879:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826993.376882:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826993.376885:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564092032:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826993.376890:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564092160:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826993.376909:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826993.376919:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826993/real 1787826993] req@ffff976889cfa1c0 x1874670564092160/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826993.376930:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564092160/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826993.376937:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826993.376940:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826993.376942:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564092160:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787826993.376948:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564092288:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826993.376965:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826993.376976:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826993/real 1787826993] req@ffff976889cf92c0 x1874670564092288/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826993.376987:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564092288/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826993.376994:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826993.376997:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826993.377000:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564092288:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826993.630930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826993.630930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787826993.631406:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826994.398977:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826994.654950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826994.655270:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787826994.655671:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787826995.422924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826995.678964:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787826995.679112:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787826995.679269:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826996.448407:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787826996.702947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826996.702948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787826996.703254:0:29023:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 5/5 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1787826996.703257:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 5 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826996.703766:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826996.703769:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826996.703771:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787826996.703802:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab193c lrc: 2/0,0 mode: CR/CR res: [0x240000404:0x14:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b83cda expref: -99 pid: 56198 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787826996.703853:0:42221:0:(vvp_object.c:117:vvp_conf_set()) [0x240000404:0x14:0x0]: losing layout lock 00010000:00010000:1.0:1787826996.703871:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1935 lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x22153cd396b83ccc expref: -99 pid: 56198 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826996.703891:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976888fb9200/0x8e7c99ee62ab09f3 lrc: 2/0,0 mode: PR/PR res: [0x240000405:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x22153cd396b8130b expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826996.703910:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff9768853fdc00/0x8e7c99ee62ab08b8 lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b80f76 expref: -99 pid: 54274 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787826996.703920:0:42221:0:(vvp_object.c:117:vvp_conf_set()) [0x240000404:0x11:0x0]: losing layout lock 00010000:00010000:1.0:1787826996.703944:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff9768853fd200/0x8e7c99ee62ab07b5 lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x10:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x22153cd396b80cba expref: -99 pid: 54253 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826996.704047:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab193c lrc: 1/0,0 mode: --/CR res: [0x240000404:0x14:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b83cda expref: -99 pid: 56198 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826996.704055:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1935 lrc: 1/0,0 mode: --/PR res: [0x240000404:0x14:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b83ccc expref: -99 pid: 56198 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826996.704062:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976888fb9200/0x8e7c99ee62ab09f3 lrc: 1/0,0 mode: --/PR res: [0x240000405:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b8130b expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826996.704069:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff9768853fdc00/0x8e7c99ee62ab08b8 lrc: 1/0,0 mode: --/PR res: [0x240000404:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b80f76 expref: -99 pid: 54274 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826996.704075:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff9768853fd200/0x8e7c99ee62ab07b5 lrc: 1/0,0 mode: --/PR res: [0x240000404:0x10:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b80cba expref: -99 pid: 54253 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826996.704081:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 5 locks packed 00010000:00010000:1.0:1787826996.704107:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976883515a00/0x8e7c99ee62ab193c lrc: 0/0,0 mode: --/CR res: [0x240000404:0x14:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b83cda expref: -99 pid: 56198 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826996.704121:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff97688925d600/0x8e7c99ee62ab1935 lrc: 0/0,0 mode: --/PR res: [0x240000404:0x14:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b83ccc expref: -99 pid: 56198 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826996.704133:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff976888fb9200/0x8e7c99ee62ab09f3 lrc: 0/0,0 mode: --/PR res: [0x240000405:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b8130b expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00080000:0.0:1787826996.704136:0:2401:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff976889efd680 x1874670564092416/t0(0) o103->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:17/18 lens 352/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1787826996.704152:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff9768853fdc00/0x8e7c99ee62ab08b8 lrc: 0/0,0 mode: --/PR res: [0x240000404:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b80f76 expref: -99 pid: 54274 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787826996.704171:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9768877a4800 lock: ffff9768853fd200/0x8e7c99ee62ab07b5 lrc: 0/0,0 mode: --/PR res: [0x240000404:0x10:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b80cba expref: -99 pid: 54253 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787826996.704183:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787826996.704185:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787826996.704188:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826997.471093:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826997.726916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787826997.726932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787826997.726979:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787826998.495023:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787826998.495151:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826998.495158:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826998.495170:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826998.495180:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1938 00000100:00080000:3.0:1787826998.495190:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826998.495198:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787826998.495245:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564092544/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826998.495269:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826998.495273:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826998.495275:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826998.495279:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1938 00000100:00080000:3.0:1787826998.495283:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826998.495287:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826998.495298:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826998.495311:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564092672/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826998.495321:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826998.495324:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826998.495326:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826998.495329:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1938 00000100:00100000:1.0:1787826998.495342:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564092544:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787826998.495344:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826998.495351:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826998.495360:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826998.495372:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564092800/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826998.495380:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826998.495382:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826998.495383:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826998.495386:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1938 00000100:00080000:3.0:1787826998.495389:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826998.495392:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826998.495395:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787826998.495404:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564092928/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787826998.495405:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787826998.495413:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826998.495416:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826998.495418:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826998.495421:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1938 00000100:00080000:3.0:1787826998.495423:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826998.495426:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787826998.495428:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826998.495435:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564093056/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787826998.495452:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826998/real 1787826998] req@ffff97688cb5c780 x1874670564092544/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826998.495469:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564092544/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826998.495478:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826998.495481:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826998.495485:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564092544:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787826998.495506:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564092672:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826998.495528:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826998.495540:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826998/real 1787826998] req@ffff97688cb5e1c0 x1874670564092672/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826998.495552:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564092672/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826998.495566:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826998.495569:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826998.495648:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564092672:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826998.495654:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564092800:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787826998.495684:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826998.495693:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826998/real 1787826998] req@ffff97688cb5c3c0 x1874670564092800/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826998.495700:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564092800/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826998.495705:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826998.495707:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826998.495709:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564092800:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787826998.495712:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564092928:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826998.495723:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826998.495728:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826998/real 1787826998] req@ffff97688cb5f840 x1874670564092928/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826998.495734:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564092928/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826998.495738:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826998.495740:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826998.495742:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564092928:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787826998.495745:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564093056:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787826998.495753:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787826998.495758:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787826998/real 1787826998] req@ffff97688cb5cf00 x1874670564093056/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826998.495763:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564093056/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787826998.495883:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787826998.495887:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787826998.495890:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564093056:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787826998.750928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787826998.750928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787826998.750970:0:29023:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 4/4 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1787826998.750994:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 4 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826998.751003:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826998.751006:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400020000 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826998.751044:0:42221:0:(vvp_io.c:1889:vvp_io_init()) [0x200000405:0x44:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787826998.751055:0:42221:0:(vvp_io.c:304:vvp_io_fini()) [0x200000405:0x44:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1787826998.751077:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003286d8a9/0x8e7c99ee62ab09c2 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x104:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x28400020000 nid: local remote: 0x22153cd396b8125c expref: -99 pid: 54857 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826998.751086:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000fa80ca9c/0x8e7c99ee62ab08c6 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x25:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2097152->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x22153cd396b80f8b expref: -99 pid: 54274 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826998.751093:0:42221:0:(vvp_io.c:1889:vvp_io_init()) [0x240000404:0x11:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787826998.751097:0:42221:0:(vvp_io.c:304:vvp_io_fini()) [0x240000404:0x11:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1787826998.751103:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000089bf8109/0x8e7c99ee62ab08bf lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x24:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->2097151) gid 0 flags: 0x800028400000000 nid: local remote: 0x22153cd396b80f84 expref: -99 pid: 54274 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826998.751108:0:42221:0:(vvp_io.c:1889:vvp_io_init()) [0x240000404:0x11:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787826998.751110:0:42221:0:(vvp_io.c:304:vvp_io_fini()) [0x240000404:0x11:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1787826998.751138:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 1/0,0 mode: --/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400020020 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826998.751143:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003286d8a9/0x8e7c99ee62ab09c2 lrc: 1/0,0 mode: --/PW res: [0x2c0000401:0x104:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4829400020020 nid: local remote: 0x22153cd396b8125c expref: -99 pid: 54857 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826998.751147:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000fa80ca9c/0x8e7c99ee62ab08c6 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x25:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2097152->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x22153cd396b80f8b expref: -99 pid: 54274 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826998.751155:0:42221:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000089bf8109/0x8e7c99ee62ab08bf lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x24:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->2097151) gid 0 flags: 0x804869400000020 nid: local remote: 0x22153cd396b80f84 expref: -99 pid: 54274 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826998.751159:0:42221:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 4 locks packed 00010000:00010000:0.0:1787826998.751196:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000f75aa4c5/0x8e7c99ee62ab1894 lrc: 0/0,0 mode: --/PW res: [0x2c0000401:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400020020 nid: local remote: 0x22153cd396b83a79 expref: -99 pid: 55751 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826998.751228:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 000000003286d8a9/0x8e7c99ee62ab09c2 lrc: 0/0,0 mode: --/PW res: [0x2c0000401:0x104:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4829400020020 nid: local remote: 0x22153cd396b8125c expref: -99 pid: 54857 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826998.751241:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 00000000fa80ca9c/0x8e7c99ee62ab08c6 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x25:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2097152->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x22153cd396b80f8b expref: -99 pid: 54274 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826998.751258:0:42221:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9768877a4800 lock: 0000000089bf8109/0x8e7c99ee62ab08bf lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x24:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->2097151) gid 0 flags: 0x804869400000020 nid: local remote: 0x22153cd396b80f84 expref: -99 pid: 54274 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826998.751272:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826998.751274:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826998.751276:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787826998.751282:0:2402:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff976882aef840 x1874670564093184/t0(0) o103->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:17/18 lens 344/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787826999.518996:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787826999.774926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787826999.774956:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826999.775006:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827000.542954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827000.799103:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827000.799288:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827000.799349:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827001.566938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827001.822921:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827001.822938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827001.822987:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827002.590995:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827002.846968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827002.846969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827002.847044:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827003.614995:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827003.615072:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827003.615079:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827003.615084:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827003.615096:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1943 00000100:00080000:1.0:1787827003.615110:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827003.615119:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827003.615173:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efd2c0 x1874670564093312/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827003.615201:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827003.615205:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827003.615207:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827003.615212:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1943 00000100:00080000:1.0:1787827003.615217:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827003.615223:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827003.615225:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564093312:192.168.203.139@tcp:250:kworker.0 00000008:00100000:1.0:1787827003.615235:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827003.615250:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564093440/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827003.615273:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827003.615277:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827003.615279:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827003.615283:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1943 00000100:00080000:1.0:1787827003.615288:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827003.615293:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827003.615298:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827003.615311:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564093568/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827003.615320:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787827003.615325:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827003.615330:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827003.615332:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827003.615337:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1943 00000100:00080000:1.0:1787827003.615341:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827003.615345:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827003.615351:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827003.615363:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564093696/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827003.615371:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827003.615375:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827003.615376:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827003.615380:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1943 00000100:00080000:1.0:1787827003.615384:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827003.615388:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827003.615393:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827003.615399:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827003/real 1787827003] req@ffff976889efd2c0 x1874670564093312/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827003.615406:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564093824/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827003.615431:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efd2c0 x1874670564093312/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827003.615445:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827003.615451:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827003.615457:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564093312:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827003.615488:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564093440:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827003.615516:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827003.615531:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827003/real 1787827003] req@ffff976889eff480 x1874670564093440/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827003.615544:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564093440/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827003.615552:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827003.615555:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827003.615572:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564093440:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827003.615579:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564093568:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827003.615600:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827003.615612:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827003/real 1787827003] req@ffff976889efc780 x1874670564093568/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827003.615639:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564093568/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827003.615647:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827003.615650:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827003.615653:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564093568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827003.615658:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564093696:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827003.615681:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827003.615692:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827003/real 1787827003] req@ffff976889efe1c0 x1874670564093696/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827003.615707:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564093696/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827003.615715:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827003.615718:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827003.615721:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564093696:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827003.615726:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564093824:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827003.615756:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827003.615769:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827003/real 1787827003] req@ffff976889efe580 x1874670564093824/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827003.615782:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564093824/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827003.615790:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827003.615794:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827003.615798:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564093824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827003.870985:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827003.871065:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827003.871065:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827004.638949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787827004.894955:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827004.895036:0:42221:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827004.895060:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827005.663276:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827005.919056:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827005.919103:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827005.919250:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827006.686999:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827006.942939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827006.943085:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827006.943109:0:29023:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 2/2 locks to regular blp list, 1 blwis in pool 00010000:00010000:3.0:1787827006.943130:0:42221:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 2 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827006.943138:0:42221:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab1943 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800028480000000 nid: local remote: 0x22153cd396b83ce1 expref: -99 pid: 56198 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827006.943149:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827007.710948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827007.966942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827007.966963:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827007.967180:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827008.734936:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827008.734946:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827008.734950:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827008.734959:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1948 00000100:00080000:1.0:1787827008.734968:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827008.734976:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827008.735009:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564093952/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827008.735036:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827008.735040:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827008.735042:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827008.735046:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1948 00000100:00080000:1.0:1787827008.735050:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827008.735054:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827008.735065:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827008.735080:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564094080/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827008.735097:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827008.735102:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787827008.735107:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827008.735107:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827008.735107:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564093952:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827008.735123:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1948 00000100:00080000:1.0:1787827008.735127:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827008.735142:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827008.735148:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827008.735164:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564094208/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827008.735174:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827008.735178:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827008.735180:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827008.735187:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1948 00000800:00000100:2.0:1787827008.735190:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787827008.735193:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827008.735198:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827008.735203:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827008.735215:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564094336/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827008.735224:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827008.735228:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827008.735230:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827008.735234:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1948 00000100:00080000:1.0:1787827008.735237:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827008.735239:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827008/real 1787827008] req@ffff976889efcb40 x1874670564093952/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827008.735243:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827008.735251:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827008.735263:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564093952/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827008.735264:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564094464/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827008.735295:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827008.735301:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827008.735305:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564093952:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827008.735332:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564094080:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827008.735355:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827008.735367:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827008/real 1787827008] req@ffff976889efe580 x1874670564094080/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827008.735379:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564094080/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827008.735386:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827008.735390:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827008.735393:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564094080:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827008.735399:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564094208:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827008.735418:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827008.735429:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827008/real 1787827008] req@ffff976889efe1c0 x1874670564094208/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827008.735446:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564094208/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827008.735454:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827008.735457:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827008.735459:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564094208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827008.735465:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564094336:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827008.735487:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827008.735498:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827008/real 1787827008] req@ffff976889efc780 x1874670564094336/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827008.735508:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564094336/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827008.735516:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827008.735519:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827008.735522:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564094336:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827008.735527:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564094464:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827008.735545:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827008.735555:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827008/real 1787827008] req@ffff976889eff480 x1874670564094464/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827008.735566:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564094464/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827008.735578:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827008.735581:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827008.735584:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564094464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827008.990965:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827008.990972:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827008.991071:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827009.758987:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827010.014974:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827010.014995:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827010.015016:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827010.782932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827011.038915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827011.038955:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827011.040529:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827011.806985:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827012.062925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827012.062987:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827012.063059:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827012.830983:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827013.086964:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827013.086964:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827013.087058:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827013.854915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827013.854931:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827013.854936:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827013.854940:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827013.854947:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1953 00000100:00080000:2.0:1787827013.854957:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827013.854964:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827013.854999:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564094592/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827013.855020:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827013.855022:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827013.855024:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827013.855026:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1953 00000100:00080000:2.0:1787827013.855029:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827013.855031:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827013.855039:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827013.855049:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564094720/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827013.855057:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827013.855060:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787827013.855061:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564094592:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787827013.855062:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827013.855068:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1953 00000100:00080000:2.0:1787827013.855072:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827013.855074:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827013.855077:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827013.855086:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564094848/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827013.855106:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827013.855109:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827013.855110:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827013.855113:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1953 00000100:00080000:2.0:1787827013.855116:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827013.855119:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827013.855122:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827013.855131:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564094976/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827013.855138:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787827013.855140:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827013.855144:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827013.855145:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827013.855147:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1953 00000100:00080000:2.0:1787827013.855150:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827013.855152:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827013.855155:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827013.855161:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564095104/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827013.855182:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827013/real 1787827013] req@ffff976889cfad00 x1874670564094592/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827013.855205:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564094592/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827013.855214:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827013.855218:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827013.855222:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564094592:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827013.855244:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564094720:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827013.855267:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827013.855280:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827013/real 1787827013] req@ffff976889cf92c0 x1874670564094720/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827013.855292:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564094720/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827013.855299:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827013.855302:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827013.855305:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564094720:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827013.855311:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564094848:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827013.855330:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827013.855341:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827013/real 1787827013] req@ffff976889cfa1c0 x1874670564094848/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827013.855352:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564094848/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827013.855364:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827013.855367:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827013.855371:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564094848:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827013.855376:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564094976:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827013.855395:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827013.855406:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827013/real 1787827013] req@ffff976889cfa580 x1874670564094976/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827013.855416:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564094976/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827013.855423:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827013.855426:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827013.855428:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564094976:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827013.855433:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564095104:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827013.855449:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827013.855459:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827013/real 1787827013] req@ffff976889cfb480 x1874670564095104/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827013.855470:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564095104/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827013.855477:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827013.855480:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827013.855484:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564095104:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827014.110937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827014.110945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827014.111037:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827014.878932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827015.134964:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827015.135063:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827015.136338:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827015.902923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827016.158929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827016.158930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827016.158959:0:29023:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 4/4 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1787827016.159003:0:56500:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 4 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827016.159305:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827016.159308:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827016.159310:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827016.159332:0:56500:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18f6 lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x22153cd396b83bf3 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827016.159367:0:56500:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab18e1 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b83b83 expref: -99 pid: 55761 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827016.159378:0:56500:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x1d:0x0]: losing layout lock 00010000:00010000:1.0:1787827016.159401:0:56500:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18da lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x22153cd396b83b60 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827016.159410:0:56500:0:(vvp_object.c:117:vvp_conf_set()) [0x200000405:0x1d:0x0]: losing layout lock 00010000:00010000:1.0:1787827016.159417:0:56500:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9c00/0x8e7c99ee62ab09e5 lrc: 2/0,0 mode: PR/PR res: [0x200000405:0x1c:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x22153cd396b812ef expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827016.159433:0:56500:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9600/0x8e7c99ee62ab0c1c lrc: 2/0,0 mode: CR/CR res: [0x200000405:0x1c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x22153cd396b818c9 expref: -99 pid: 55483 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00010000:00010000:1.0:1787827016.159485:0:56500:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18f6 lrc: 1/0,0 mode: --/PR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b83bf3 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827016.159493:0:56500:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab18e1 lrc: 1/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b83b83 expref: -99 pid: 55761 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827016.159500:0:56500:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18da lrc: 1/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b83b60 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827016.159511:0:56500:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9c00/0x8e7c99ee62ab09e5 lrc: 1/0,0 mode: --/PR res: [0x200000405:0x1c:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b812ef expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827016.159517:0:56500:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 4 locks packed 00010000:00010000:1.0:1787827016.159541:0:56500:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925d800/0x8e7c99ee62ab18f6 lrc: 0/0,0 mode: --/PR res: [0x240000404:0x13:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b83bf3 expref: -99 pid: 56417 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827016.159555:0:56500:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff97688925ce00/0x8e7c99ee62ab18e1 lrc: 0/0,0 mode: --/PR res: [0x200000405:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b83b83 expref: -99 pid: 55761 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827016.159565:0:56500:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976883514400/0x8e7c99ee62ab18da lrc: 0/0,0 mode: --/CR res: [0x200000405:0x1d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x22153cd396b83b60 expref: -99 pid: 55759 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827016.159577:0:56500:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9768877a4800 lock: ffff976888fb9c00/0x8e7c99ee62ab09e5 lrc: 0/0,0 mode: --/PR res: [0x200000405:0x1c:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x22153cd396b812ef expref: -99 pid: 55443 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00080000:0.0:1787827016.159582:0:2401:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff976889efd2c0 x1874670564095232/t0(0) o103->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:17/18 lens 344/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1787827016.159601:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827016.159604:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827016.159607:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827016.926919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827017.182928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827017.182928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827017.182956:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827017.950943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827018.206929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827018.206929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827018.207121:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827018.974928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827018.974930:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827018.974938:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827018.974942:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827018.974951:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1958 00000100:00080000:1.0:1787827018.974961:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827018.974968:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827018.975004:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564095360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827018.975030:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827018.975033:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827018.975035:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827018.975040:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1958 00000100:00080000:1.0:1787827018.975043:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827018.975047:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827018.975057:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827018.975075:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564095360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827018.975082:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564095488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827018.975093:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827018.975097:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827018.975098:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827018.975102:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1958 00000100:00080000:1.0:1787827018.975106:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827018.975109:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827018.975114:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827018.975126:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564095616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827018.975136:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787827018.975141:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787827018.975142:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827018.975147:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827018.975152:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1958 00000100:00080000:1.0:1787827018.975157:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827018.975161:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827018.975166:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827018.975178:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564095744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827018.975185:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827018/real 1787827018] req@ffff976889efc3c0 x1874670564095360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827018.975189:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827018.975197:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827018.975200:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827018.975206:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1958 00000100:00080000:1.0:1787827018.975210:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827018.975213:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564095360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827018.975217:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827018.975224:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827018.975230:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787827018.975239:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564095872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827018.975239:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827018.975248:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564095360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827018.975280:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564095488:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827018.975301:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827018.975313:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827018/real 1787827018] req@ffff976889eff0c0 x1874670564095488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827018.975330:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564095488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827018.975338:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827018.975341:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827018.975344:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564095488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827018.975349:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564095616:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827018.975366:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827018.975375:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827018/real 1787827018] req@ffff976889effc00 x1874670564095616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827018.975385:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564095616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827018.975393:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827018.975396:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827018.975399:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564095616:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827018.975404:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564095744:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827018.975422:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827018.975432:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827018/real 1787827018] req@ffff976889efed00 x1874670564095744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827018.975442:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564095744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827018.975455:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827018.975458:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827018.975460:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564095744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827018.975466:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564095872:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827018.975482:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827018.975491:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827018/real 1787827018] req@ffff976889efe940 x1874670564095872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827018.975501:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564095872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827018.975508:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827018.975510:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827018.975513:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564095872:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827019.230942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827019.230956:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827019.230980:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827019.999314:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787827020.255956:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827020.256102:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827020.256128:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827021.022917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787827021.278956:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827021.278970:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827021.279074:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827022.046966:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827022.302954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827022.303009:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827022.303012:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827023.070972:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827023.326958:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827023.326962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827023.327047:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827024.094897:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827024.094908:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827024.094911:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827024.094921:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1963 00000100:00080000:1.0:1787827024.094929:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827024.094935:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827024.094938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827024.094965:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564096000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827024.094994:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827024.094997:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827024.094999:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827024.095009:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1963 00000100:00080000:1.0:1787827024.095012:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827024.095017:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827024.095037:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827024.095052:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564096128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827024.095062:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827024.095065:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827024.095067:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827024.095073:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1963 00000100:00080000:1.0:1787827024.095077:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827024.095084:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564096000:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827024.095085:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827024.095095:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827024.095114:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564096256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827024.095123:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827024.095125:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827024.095126:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827024.095130:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1963 00000100:00080000:1.0:1787827024.095132:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827024.095135:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827024.095139:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827024.095150:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564096384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827024.095160:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827024.095163:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827024.095166:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827024.095170:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1963 00000100:00080000:1.0:1787827024.095176:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827024.095181:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827024.095187:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827024.095204:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827024.095210:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564096512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827024.095272:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827024/real 1787827024] req@ffff976889efe940 x1874670564096000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827024.095305:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564096000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827024.095317:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827024.095322:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827024.095327:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564096000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827024.095337:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564096128:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827024.095374:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827024.095388:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827024/real 1787827024] req@ffff976889efed00 x1874670564096128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827024.095399:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564096128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827024.095408:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827024.095411:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827024.095414:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564096128:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827024.095445:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564096256:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827024.095466:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827024.095477:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827024/real 1787827024] req@ffff976889effc00 x1874670564096256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827024.095490:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564096256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827024.095503:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827024.095506:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827024.095509:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564096256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827024.095515:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564096384:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827024.095535:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827024.095545:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827024/real 1787827024] req@ffff976889eff0c0 x1874670564096384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827024.095556:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564096384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827024.095564:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827024.095568:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827024.095571:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564096384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827024.095578:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564096512:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827024.095597:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827024.095609:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827024/real 1787827024] req@ffff976889efc3c0 x1874670564096512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827024.095619:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564096512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827024.095627:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827024.095630:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827024.095633:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564096512:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827024.350924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827024.350959:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827024.350983:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827025.119089:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827025.374942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827025.374953:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827025.374991:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827026.142976:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827026.399090:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827026.399111:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827026.399124:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827027.166971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787827027.422929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827027.422956:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827027.423027:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827028.190987:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827028.446975:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827028.446981:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827028.447070:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827029.214924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827029.215067:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827029.215074:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827029.215078:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827029.215090:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1969 00000100:00080000:3.0:1787827029.215104:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827029.215113:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827029.215166:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564096640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827029.215193:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827029.215196:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827029.215201:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827029.215205:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1969 00000100:00080000:3.0:1787827029.215208:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827029.215212:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827029.215221:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827029.215233:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564096768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827029.215242:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827029.215245:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827029.215248:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827029.215254:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1969 00000100:00080000:3.0:1787827029.215259:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827029.215263:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827029.215268:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827029.215277:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564096896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827029.215287:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827029.215290:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827029.215292:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827029.215301:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1969 00000100:00100000:2.0:1787827029.215304:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564096640:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827029.215308:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827029.215323:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827029.215330:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827029.215346:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564097024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827029.215355:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827029.215358:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827029.215360:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827029.215364:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1969 00000100:00080000:3.0:1787827029.215367:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827029.215371:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827029.215375:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827029.215385:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564097152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827029.215581:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827029.215640:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827029/real 1787827029] req@ffff97688cb5d2c0 x1874670564096640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827029.215726:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564096640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827029.215739:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827029.215744:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827029.215748:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564096640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827029.215764:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564096768:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827029.215792:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827029.215806:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827029/real 1787827029] req@ffff97688cb5cf00 x1874670564096768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827029.215935:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564096768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827029.215945:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827029.215957:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827029.215962:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564096768:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827029.216017:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564096896:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827029.216041:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827029.216053:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827029/real 1787827029] req@ffff97688cb5f840 x1874670564096896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827029.216064:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564096896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827029.216077:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827029.216081:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827029.216084:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564096896:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827029.216092:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564097024:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827029.216112:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827029.216124:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827029/real 1787827029] req@ffff97688cb5c3c0 x1874670564097024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827029.216135:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564097024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827029.216143:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827029.216474:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827029.216481:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564097024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827029.216493:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564097152:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827029.216554:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827029.216581:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827029/real 1787827029] req@ffff97688cb5e1c0 x1874670564097152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827029.216598:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564097152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827029.216608:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827029.216612:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827029.216615:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564097152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827029.471088:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827029.471132:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827029.471616:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827030.238932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827030.494913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827030.494914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827030.494938:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827031.262916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827031.518914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827031.518990:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827031.518994:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827032.286893:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827032.542939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827032.543020:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827032.543307:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827033.311002:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827033.566940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827033.567090:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827033.567129:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827034.334943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827034.334969:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827034.334976:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827034.334980:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827034.334990:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1974 00000100:00080000:1.0:1787827034.335001:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827034.335009:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827034.335049:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564097280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827034.335079:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827034.335082:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827034.335084:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827034.335088:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1974 00000100:00080000:1.0:1787827034.335092:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827034.335095:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827034.335106:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827034.335131:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564097408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827034.335143:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827034.335148:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827034.335150:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827034.335157:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1974 00000100:00100000:2.0:1787827034.335159:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564097280:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827034.335162:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827034.335174:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827034.335180:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827034.335195:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564097536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827034.335205:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827034.335208:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827034.335210:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827034.335215:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1974 00000100:00080000:1.0:1787827034.335219:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827034.335223:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827034.335229:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827034.335240:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564097664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827034.335249:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827034.335252:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827034.335254:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827034.335258:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1974 00000800:00000100:2.0:1787827034.335267:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787827034.335268:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827034.335276:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827034.335282:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827034.335295:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564097792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827034.335336:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827034/real 1787827034] req@ffff976889eff480 x1874670564097280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827034.335358:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564097280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827034.335371:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827034.335375:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827034.335379:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564097280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827034.335411:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564097408:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827034.335438:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827034.335453:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827034/real 1787827034] req@ffff976889efc780 x1874670564097408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827034.335464:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564097408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827034.335482:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827034.335486:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827034.335489:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564097408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827034.335496:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564097536:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827034.335517:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827034.335528:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827034/real 1787827034] req@ffff976889efe1c0 x1874670564097536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827034.335540:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564097536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827034.335547:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827034.335551:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827034.335554:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564097536:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827034.335575:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564097664:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827034.335597:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827034.335608:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827034/real 1787827034] req@ffff976889efe580 x1874670564097664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827034.335619:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564097664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827034.335628:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827034.335632:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827034.335635:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564097664:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827034.335641:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564097792:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827034.335662:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827034.335674:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827034/real 1787827034] req@ffff976889efcb40 x1874670564097792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827034.335685:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564097792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827034.335693:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827034.335696:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827034.335699:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564097792:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827034.590933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827034.590933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827034.591013:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827035.358952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827035.614938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827035.614982:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827035.615064:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827036.382972:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827036.638927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827036.638969:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827036.639040:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827037.406960:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827037.662938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827037.662941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827037.662986:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827038.430914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827038.686902:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827038.686931:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827038.687012:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00080000:0.0:1787827039.455142:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827039.455149:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827039.455154:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827039.455163:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1979 00000100:00080000:0.0:1787827039.455174:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827039.455181:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827039.455211:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564097920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827039.455232:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827039.455236:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827039.455238:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827039.455242:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1979 00000100:00080000:0.0:1787827039.455245:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827039.455249:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827039.455254:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564097920:192.168.203.139@tcp:250:kworker.0 00000008:00100000:0.0:1787827039.455260:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827039.455276:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564098048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827039.455288:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827039.455291:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827039.455293:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827039.455297:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1979 00000100:00080000:0.0:1787827039.455301:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827039.455305:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827039.455309:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827039.455332:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564098176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827039.455332:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787827039.455344:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827039.455347:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827039.455349:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827039.455354:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1979 00000100:00080000:0.0:1787827039.455358:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827039.455361:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827039.455367:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827039.455377:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827039/real 1787827039] req@ffff976882aed680 x1874670564097920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827039.455381:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564098304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827039.455403:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827039.455406:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827039.455407:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827039.455411:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564097920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827039.455415:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1979 00000100:00080000:0.0:1787827039.455421:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827039.455430:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827039.455434:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:3.0:1787827039.455436:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827039.455443:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564097920:192.168.203.139@tcp:250:kworker.0 00000008:00100000:0.0:1787827039.455444:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827039.455461:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564098432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827039.455474:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564098048:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827039.455504:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827039.455519:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827039/real 1787827039] req@ffff976882aec3c0 x1874670564098048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827039.455532:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564098048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827039.455541:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827039.455544:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827039.455548:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564098048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827039.455554:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564098176:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827039.455579:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827039.455592:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827039/real 1787827039] req@ffff976882aeed00 x1874670564098176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827039.455605:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564098176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827039.455613:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827039.455616:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827039.455619:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564098176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827039.455625:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564098304:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827039.455646:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827039.455647:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787827039.455719:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827039/real 1787827039] req@ffff976882aee580 x1874670564098304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827039.455731:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564098304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827039.455739:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827039.455742:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827039.455746:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564098304:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827039.455767:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564098432:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827039.455789:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827039.455803:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827039/real 1787827039] req@ffff976882aee940 x1874670564098432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827039.455849:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564098432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827039.455858:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827039.455862:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827039.455867:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564098432:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827039.710935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827039.710935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827039.710983:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827040.478954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827040.734932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827040.734963:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827040.734987:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827041.502956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827041.758939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827041.758939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827041.759027:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827042.526929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827042.782907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827042.782907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827042.783947:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827043.550948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827043.806935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827043.806935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827043.806997:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827044.574953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827044.574993:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827044.575003:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827044.575007:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827044.575017:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1984 00000100:00080000:2.0:1787827044.575026:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827044.575033:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827044.575069:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564098560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827044.575094:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827044.575098:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827044.575100:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827044.575104:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1984 00000100:00080000:2.0:1787827044.575108:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827044.575112:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827044.575121:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827044.575131:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564098560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827044.575135:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564098688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827044.575146:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827044.575150:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827044.575152:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827044.575156:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1984 00000100:00080000:2.0:1787827044.575160:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827044.575163:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827044.575176:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827044.575191:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564098816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827044.575206:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827044.575209:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827044.575211:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827044.575216:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1984 00000100:00080000:2.0:1787827044.575220:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827044.575223:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827044.575227:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827044.575232:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564098688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827044.575241:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564098944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827044.575251:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827044.575254:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827044.575256:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827044.575259:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1984 00000100:00080000:2.0:1787827044.575263:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827044.575268:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827044.575269:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564098816:192.168.203.139@tcp:38:kworker.0 00000008:00100000:2.0:1787827044.575273:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827044.575287:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564099072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827044.575305:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564098944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827044.575332:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564099072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827044.830936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827044.830944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827044.830961:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827045.599000:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827045.854942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827045.855000:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827045.855066:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827046.623003:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827046.878977:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827046.879026:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827046.879072:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827047.647085:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:2.0:1787827047.647445:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:2.0:1787827047.647487:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787827047.647504:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787827047.647550:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787827047.647559:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787827047.647567:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787827047.647574:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:3.0:1787827047.647728:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827044/real 1787827047] req@ffff976889cf9e00 x1874670564098944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827047.647752:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564098944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827047.647763:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827047.647768:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827047.647773:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564098944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827047.647780:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827044/real 1787827047] req@ffff976889cfa940 x1874670564099072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827047.647887:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564099072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827047.647899:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827047.647902:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827047.647906:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564099072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827047.647913:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827044/real 1787827047] req@ffff976889cfbc00 x1874670564098688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827047.647926:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564098688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827047.647933:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827047.647937:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827047.647940:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564098688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827047.647950:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827044/real 1787827047] req@ffff976889cf8000 x1874670564098816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827047.647961:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564098816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827047.647968:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827047.647971:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827047.647973:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564098816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827047.647979:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827044/real 1787827047] req@ffff976889cfb0c0 x1874670564098560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827047.647990:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564098560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827047.647998:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827047.648011:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827047.648014:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564098560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827047.902940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827047.902940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827047.903033:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827048.671483:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827048.926949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827048.926949:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827048.927005:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827049.695016:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827049.695026:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827049.695031:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827049.695036:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827049.695044:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1989 00000100:00080000:0.0:1787827049.695057:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827049.695065:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827049.695099:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564099200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827049.695125:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827049.695129:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827049.695131:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827049.695134:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1989 00000100:00080000:0.0:1787827049.695137:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827049.695140:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827049.695150:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827049.695164:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564099328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827049.695172:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564099200:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787827049.695177:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827049.695182:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827049.695183:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827049.695188:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1989 00000100:00080000:0.0:1787827049.695191:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827049.695194:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827049.695199:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827049.695212:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564099456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827049.695220:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827049.695224:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827049.695226:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827049.695230:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1989 00000100:00080000:0.0:1787827049.695234:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827049.695237:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827049.695243:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827049.695255:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564099584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827049.695263:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827049.695266:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827049.695267:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827049.695271:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1989 00000100:00080000:0.0:1787827049.695274:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827049.695277:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827049.695281:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827049.695290:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564099712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827049.695547:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827049.695596:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827049/real 1787827049] req@ffff976882aef0c0 x1874670564099200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827049.695614:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564099200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827049.695625:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827049.695630:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827049.695633:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564099200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827049.695666:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564099328:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827049.697623:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827049.697656:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827049/real 1787827049] req@ffff976882aee940 x1874670564099328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827049.697910:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564099328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827049.697922:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827049.697927:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827049.697931:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564099328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827049.697942:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564099456:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827049.697978:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827049.697994:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827049/real 1787827049] req@ffff976882aee580 x1874670564099456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827049.698007:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564099456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827049.698014:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827049.698017:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827049.698020:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564099456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827049.698027:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564099584:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827049.698044:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827049.698054:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827049/real 1787827049] req@ffff976882aeed00 x1874670564099584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827049.698064:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564099584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827049.698071:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827049.698074:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827049.698077:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564099584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827049.698082:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564099712:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827049.698099:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827049.698109:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827049/real 1787827049] req@ffff976882aec3c0 x1874670564099712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827049.698130:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564099712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827049.698137:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827049.698140:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827049.698143:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564099712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827049.950930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827049.950930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827049.950986:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827050.719043:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827050.974934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827050.974934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827050.975018:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827051.742937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827051.998941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827051.998941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827051.999040:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827052.766931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827053.022949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827053.022977:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827053.022988:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827053.790981:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827054.046941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827054.046941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827054.047007:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827054.814946:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827054.814950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827054.814958:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827054.814963:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827054.814972:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1994 00000100:00080000:3.0:1787827054.814982:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827054.814989:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827054.815021:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564099840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827054.815045:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827054.815047:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827054.815048:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827054.815051:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1994 00000100:00080000:3.0:1787827054.815054:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827054.815056:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827054.815063:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827054.815073:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564099968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827054.815082:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827054.815085:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827054.815087:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827054.815090:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1994 00000100:00080000:3.0:1787827054.815093:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827054.815096:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827054.815101:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827054.815110:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564100096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827054.815119:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827054.815122:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827054.815123:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827054.815126:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1994 00000100:00080000:3.0:1787827054.815128:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827054.815130:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827054.815134:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827054.815141:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564100224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827054.815146:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827054.815148:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827054.815149:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827054.815152:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1994 00000100:00080000:3.0:1787827054.815155:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827054.815158:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827054.815162:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827054.815172:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564100352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827054.815183:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564099840:192.168.203.139@tcp:250:kworker.0 00000800:00000100:2.0:1787827054.815252:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827054.815298:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827054/real 1787827054] req@ffff97688cb5c780 x1874670564099840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827054.815314:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564099840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827054.815324:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827054.815327:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827054.815331:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564099840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827054.815338:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564099968:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827054.815360:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827054.815372:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827054/real 1787827054] req@ffff97688cb5cb40 x1874670564099968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827054.815383:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564099968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827054.815390:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827054.815392:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827054.815395:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564099968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827054.815400:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564100096:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827054.815418:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827054.815428:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827054/real 1787827054] req@ffff97688cb5c000 x1874670564100096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827054.815440:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564100096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827054.815448:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827054.815451:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827054.815454:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564100096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827054.815459:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564100224:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827054.815478:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827054.815488:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827054/real 1787827054] req@ffff97688cb5d680 x1874670564100224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827054.815498:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564100224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827054.815510:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827054.815512:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827054.815515:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564100224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827054.815540:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564100352:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827054.815558:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827054.815567:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827054/real 1787827054] req@ffff97688cb5f0c0 x1874670564100352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827054.815577:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564100352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827054.815584:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827054.815587:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827054.815589:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564100352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827055.071008:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827055.071008:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827055.071085:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827055.839080:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827056.095017:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827056.095125:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827056.095251:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827056.863017:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827057.118945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827057.118989:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827057.119017:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827057.886930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827058.142894:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827058.142897:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827058.143845:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827058.911003:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827059.166918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827059.166946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827059.166977:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827059.935110:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827059.935201:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827059.935210:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827059.935215:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827059.935224:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 1999 00000100:00080000:3.0:1787827059.935235:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827059.935242:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827059.935279:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564100480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827059.935311:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827059.935316:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827059.935317:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827059.935321:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1999 00000100:00080000:3.0:1787827059.935324:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827059.935328:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827059.935337:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827059.935351:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564100608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827059.935359:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827059.935362:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827059.935364:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827059.935367:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1999 00000100:00080000:3.0:1787827059.935370:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827059.935372:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827059.935376:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827059.935385:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564100736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827059.935392:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827059.935394:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827059.935395:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827059.935398:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1999 00000100:00080000:3.0:1787827059.935402:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827059.935405:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827059.935410:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827059.935417:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564100864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827059.935423:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827059.935425:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827059.935426:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827059.935429:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 1999 00000100:00080000:3.0:1787827059.935432:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827059.935435:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827059.935439:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827059.935447:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564100992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827059.935539:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564100480:192.168.203.139@tcp:250:kworker.0 00000800:00000100:2.0:1787827059.935598:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827059.935634:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827059/real 1787827059] req@ffff97688cb5de00 x1874670564100480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827059.935648:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564100480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827059.935658:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827059.935662:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827059.935665:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564100480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827059.935670:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564100608:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827059.935688:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827059.935698:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827059/real 1787827059] req@ffff97688cb5f0c0 x1874670564100608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827059.935709:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564100608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827059.935717:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827059.935720:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827059.935722:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564100608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827059.935727:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564100736:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827059.935740:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827059.935749:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827059/real 1787827059] req@ffff97688cb5d680 x1874670564100736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827059.935758:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564100736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827059.935766:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827059.935769:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827059.935771:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564100736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827059.935790:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564100864:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827059.935807:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827059.935816:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827059/real 1787827059] req@ffff97688cb5c000 x1874670564100864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827059.935854:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564100864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827059.935862:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827059.935865:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827059.935867:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564100864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827059.935872:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564100992:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827059.935888:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827059.935896:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827059/real 1787827059] req@ffff97688cb5cb40 x1874670564100992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827059.935906:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564100992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827059.935912:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827059.935914:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827059.935917:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564100992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827060.191016:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827060.191137:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827060.191159:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827060.958998:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827061.214942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827061.214988:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827061.215741:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827061.982911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827062.238903:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827062.238953:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827062.239487:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827063.006895:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827063.262934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827063.262934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827063.263801:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827064.030954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827064.286929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827064.287008:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827064.287033:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827065.054943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827065.055016:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827065.055021:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827065.055024:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827065.055031:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2004 00000100:00080000:1.0:1787827065.055040:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827065.055045:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827065.055077:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564101120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827065.055100:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827065.055102:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827065.055103:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827065.055106:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2004 00000100:00080000:1.0:1787827065.055108:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827065.055110:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827065.055117:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827065.055124:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564101248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827065.055130:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827065.055132:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827065.055133:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827065.055135:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2004 00000100:00080000:1.0:1787827065.055136:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827065.055138:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827065.055141:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827065.055148:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564101376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827065.055157:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827065.055160:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827065.055161:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827065.055165:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2004 00000100:00080000:1.0:1787827065.055169:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827065.055173:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827065.055179:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827065.055191:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564101504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827065.055192:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564101120:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827065.055207:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827065.055209:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827065.055211:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827065.055213:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2004 00000100:00080000:1.0:1787827065.055216:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827065.055218:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827065.055221:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827065.055230:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564101632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827065.055267:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827065.055308:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827065/real 1787827065] req@ffff976889efcb40 x1874670564101120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827065.055327:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564101120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827065.055338:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827065.055343:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827065.055346:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564101120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827065.055353:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564101248:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827065.055373:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827065.055385:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827065/real 1787827065] req@ffff976889efe580 x1874670564101248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827065.055395:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564101248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827065.055404:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827065.055407:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827065.055410:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564101248:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827065.055416:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564101376:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827065.055434:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827065.055443:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827065/real 1787827065] req@ffff976889efe1c0 x1874670564101376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827065.055455:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564101376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827065.055463:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827065.055466:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827065.055469:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564101376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827065.055493:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564101504:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827065.055512:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827065.055523:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827065/real 1787827065] req@ffff976889efc780 x1874670564101504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827065.055534:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564101504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827065.055542:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827065.055544:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827065.055548:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564101504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827065.055553:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564101632:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827065.055569:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827065.055579:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827065/real 1787827065] req@ffff976889eff480 x1874670564101632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827065.055594:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564101632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827065.055601:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827065.055604:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827065.055607:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564101632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827065.311038:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827065.311041:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827065.311089:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827066.078961:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827066.334970:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827066.335086:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827066.335100:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827067.102950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827067.359079:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827067.359156:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827067.359172:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827068.126971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827068.382952:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827068.382953:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827068.383044:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827069.153961:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827069.406953:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827069.407021:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827069.407324:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827070.174931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827070.174983:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827070.174989:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827070.174993:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827070.175002:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2010 00000100:00080000:3.0:1787827070.175012:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827070.175020:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827070.175057:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564101760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827070.175084:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827070.175087:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827070.175089:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827070.175093:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2010 00000100:00080000:3.0:1787827070.175096:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827070.175107:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827070.175116:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827070.175130:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564101888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827070.175137:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564101760:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827070.175144:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827070.175147:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827070.175149:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827070.175154:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2010 00000100:00080000:3.0:1787827070.175159:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827070.175162:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827070.175167:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827070.175179:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564102016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827070.175189:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827070.175192:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827070.175194:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827070.175198:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2010 00000100:00080000:3.0:1787827070.175202:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827070.175206:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827070.175211:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1787827070.175214:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827070.175225:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564102144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827070.175233:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827070.175236:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827070.175238:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827070.175241:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2010 00000100:00080000:3.0:1787827070.175244:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827070.175248:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827070.175253:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827070.175265:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564102272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827070.175283:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827070/real 1787827070] req@ffff97688cb5c780 x1874670564101760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827070.175301:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564101760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827070.175312:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827070.175316:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827070.175321:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564101760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827070.175343:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564101888:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827070.175364:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827070.175376:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827070/real 1787827070] req@ffff97688cb5e1c0 x1874670564101888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827070.175388:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564101888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827070.175396:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827070.175399:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827070.175402:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564101888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827070.175408:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564102016:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827070.175424:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827070.175432:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827070/real 1787827070] req@ffff97688cb5c3c0 x1874670564102016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827070.175443:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564102016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827070.175451:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827070.175454:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827070.175457:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564102016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827070.175462:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564102144:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827070.175479:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827070.175488:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827070/real 1787827070] req@ffff97688cb5f840 x1874670564102144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827070.175498:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564102144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827070.175524:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827070.175528:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827070.175531:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564102144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827070.175536:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564102272:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827070.175555:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827070.175579:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827070/real 1787827070] req@ffff97688cb5cf00 x1874670564102272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827070.175590:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564102272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827070.175597:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827070.175600:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827070.175604:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564102272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827070.430937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827070.430962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827070.431026:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827071.198895:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827071.454932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827071.454955:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827071.455083:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827072.222946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827072.478965:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827072.479015:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827072.479125:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827073.246939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827073.502932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827073.502941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827073.503077:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827074.270932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827074.526950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827074.527319:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827074.527746:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827075.294940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827075.295424:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827075.295433:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827075.295438:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827075.295450:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2015 00000100:00080000:0.0:1787827075.295471:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827075.295481:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827075.295651:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564102400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827075.295692:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827075.295696:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827075.295698:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827075.295703:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2015 00000100:00080000:0.0:1787827075.295707:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827075.295713:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827075.295725:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827075.295733:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564102400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827075.295742:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564102528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827075.295753:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827075.295757:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827075.295759:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827075.295764:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2015 00000100:00080000:0.0:1787827075.295767:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827075.295771:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827075.295776:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827075.295788:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564102656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827075.295798:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827075.295801:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827075.295803:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827075.295807:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2015 00000100:00080000:0.0:1787827075.295811:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827075.295815:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787827075.295819:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827075.295842:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827075.295855:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564102784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827075.295865:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827075.295868:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827075.295871:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827075.295875:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2015 00000100:00080000:0.0:1787827075.295880:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827075.295884:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827075.295889:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827075.295890:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827075/real 1787827075] req@ffff976882aed680 x1874670564102400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827075.295904:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564102912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827075.295929:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564102400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827075.295941:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827075.295946:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827075.295950:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564102400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827075.295997:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564102528:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827075.296018:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827075.296030:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827075/real 1787827075] req@ffff976882aec3c0 x1874670564102528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827075.296041:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564102528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827075.296050:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827075.296053:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827075.296056:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564102528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827075.296061:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564102656:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827075.296077:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827075.296085:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827075/real 1787827075] req@ffff976882aeed00 x1874670564102656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827075.296096:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564102656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827075.296111:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827075.296114:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827075.296117:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564102656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827075.296122:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564102784:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827075.296137:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827075.296145:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827075/real 1787827075] req@ffff976882aee580 x1874670564102784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827075.296156:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564102784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827075.296164:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827075.296167:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827075.296170:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564102784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827075.296176:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564102912:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827075.296190:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827075.296198:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827075/real 1787827075] req@ffff976882aee940 x1874670564102912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827075.296209:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564102912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827075.296216:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827075.296219:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827075.296223:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564102912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827075.550957:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827075.550957:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827075.551210:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827076.318987:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827076.574924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827076.574964:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827076.575020:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827077.342926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827077.598889:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827077.598904:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827077.599014:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827078.366909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827078.622924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827078.622925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827078.623070:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827079.390947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827079.646870:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827079.646955:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827079.647034:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00080000:1.0:1787827080.414985:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827080.414987:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827080.414997:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827080.415003:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827080.415015:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2020 00000100:00080000:1.0:1787827080.415029:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827080.415039:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827080.415088:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564103040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827080.415122:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827080.415126:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827080.415128:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827080.415132:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2020 00000100:00080000:1.0:1787827080.415136:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827080.415140:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827080.415161:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827080.415177:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564103168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827080.415188:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827080.415192:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787827080.415192:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564103040:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827080.415196:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827080.415204:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2020 00000100:00080000:1.0:1787827080.415207:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827080.415211:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827080.415217:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827080.415231:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564103296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827080.415240:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827080.415243:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827080.415245:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827080.415250:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2020 00000100:00080000:1.0:1787827080.415255:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827080.415258:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827080.415264:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827080.415277:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564103424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827080.415279:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787827080.415290:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827080.415294:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827080.415296:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827080.415301:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2020 00000100:00080000:1.0:1787827080.415307:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827080.415311:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827080.415317:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827080.415329:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564103552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827080.415335:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827080/real 1787827080] req@ffff976889efc3c0 x1874670564103040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827080.415354:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564103040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827080.415364:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827080.415368:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827080.415372:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564103040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827080.415400:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564103168:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827080.415431:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827080.415445:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827080/real 1787827080] req@ffff976889eff0c0 x1874670564103168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827080.415457:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564103168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827080.415466:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827080.415469:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827080.415473:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564103168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827080.415479:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564103296:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827080.415500:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827080.415511:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827080/real 1787827080] req@ffff976889effc00 x1874670564103296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827080.415523:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564103296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827080.415531:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827080.415534:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827080.415537:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564103296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827080.415543:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564103424:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827080.415563:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827080.415573:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827080/real 1787827080] req@ffff976889efed00 x1874670564103424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827080.415584:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564103424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827080.415592:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827080.415595:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827080.415598:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564103424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827080.415604:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564103552:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827080.415624:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827080.415648:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827080/real 1787827080] req@ffff976889efe940 x1874670564103552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827080.415661:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564103552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827080.415669:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827080.415672:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827080.415675:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564103552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827080.670920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827080.670945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827080.670992:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827081.438953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827081.694968:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827081.694985:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827081.695026:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827082.462920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827082.718924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827082.718984:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827082.719021:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827083.487270:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827083.742934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827083.743004:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827083.743030:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827084.510934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827084.766969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827084.767002:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827084.767074:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827085.534923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827085.534923:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827085.534933:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827085.534937:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827085.534948:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2025 00000100:00080000:1.0:1787827085.534957:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827085.534963:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827085.534999:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564103680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827085.535026:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827085.535029:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827085.535030:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827085.535033:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2025 00000100:00080000:1.0:1787827085.535036:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827085.535039:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827085.535046:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827085.535057:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564103808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827085.535066:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827085.535069:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827085.535071:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827085.535075:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2025 00000100:00100000:2.0:1787827085.535079:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564103680:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827085.535080:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827085.535088:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827085.535093:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827085.535107:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564103936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827085.535114:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827085.535116:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827085.535118:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827085.535121:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2025 00000100:00080000:1.0:1787827085.535123:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827085.535126:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827085.535129:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827085.535138:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564104064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827085.535145:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787827085.535153:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827085.535156:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827085.535158:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827085.535164:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2025 00000100:00080000:1.0:1787827085.535175:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827085.535182:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827085.535187:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827085.535189:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827085/real 1787827085] req@ffff976889efe940 x1874670564103680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827085.535201:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564104192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827085.535211:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564103680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827085.535223:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827085.535227:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827085.535231:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564103680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827085.535240:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564103808:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827085.535267:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827085.535279:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827085/real 1787827085] req@ffff976889efed00 x1874670564103808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827085.535290:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564103808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827085.535298:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827085.535302:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827085.535305:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564103808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827085.535329:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564103936:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827085.535350:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827085.535361:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827085/real 1787827085] req@ffff976889effc00 x1874670564103936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827085.535372:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564103936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827085.535380:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827085.535383:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827085.535385:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564103936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827085.535391:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564104064:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827085.535410:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827085.535420:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827085/real 1787827085] req@ffff976889eff0c0 x1874670564104064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827085.535444:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564104064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827085.535452:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827085.535455:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827085.535458:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564104064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827085.535464:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564104192:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827085.535483:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827085.535493:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827085/real 1787827085] req@ffff976889efc3c0 x1874670564104192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827085.535501:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564104192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827085.535506:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827085.535508:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827085.535510:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564104192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827085.790918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827085.790937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827085.790995:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827086.558924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827086.814926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827086.814926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827086.815041:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827087.582937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827087.838928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827087.838937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827087.839050:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827088.606931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827088.862924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827088.862924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827088.862979:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827089.630921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827089.886921:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827089.886985:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827089.887023:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00080000:0.0:1787827090.654978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827090.654978:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827090.654992:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827090.654998:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827090.655012:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2030 00000100:00080000:0.0:1787827090.655026:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827090.655035:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827090.655084:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564104320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827090.655123:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827090.655127:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827090.655129:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827090.655136:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2030 00000100:00080000:0.0:1787827090.655140:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827090.655152:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827090.655164:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827090.655182:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564104320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827090.655183:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564104448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827090.655209:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827090.655213:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827090.655215:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827090.655219:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2030 00000100:00080000:0.0:1787827090.655223:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827090.655227:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827090.655232:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827090.655267:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827090.655333:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827090/real 1787827090] req@ffff976882aef0c0 x1874670564104320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827090.655349:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564104576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827090.655353:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564104320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827090.655384:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827090.655384:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827090.655393:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827090.655398:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564104320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827090.655427:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564104448:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827090.655451:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787827090.655465:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787827090.655465:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827090/real 1787827090] req@ffff976882aee940 x1874670564104448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827090.655470:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827090.655481:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2030 00000100:00100000:2.0:1787827090.655487:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564104448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827090.655497:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827090.655498:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827090.655503:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00080000:0.0:1787827090.655508:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827090.655510:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564104448:192.168.203.139@tcp:38:kworker.0 00000008:00100000:0.0:1787827090.655517:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827090.655521:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564104576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827090.655536:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564104704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827090.655548:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827090.655552:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827090.655554:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787827090.655557:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827090.655584:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827090/real 1787827090] req@ffff976882aee580 x1874670564104576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827090.655597:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564104576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827090.655612:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827090.655615:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827090.655618:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564104576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827090.655632:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564104704:192.168.203.139@tcp:8:kworker.0 00000100:00080000:0.0:1787827090.655645:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2030 00000100:00080000:0.0:1787827090.655651:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827090.655656:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827090.655662:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827090.655668:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827090.655678:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564104832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827090.655683:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827090/real 1787827090] req@ffff976882aeed00 x1874670564104704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827090.655703:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564104704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827090.655712:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827090.655717:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827090.655721:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564104704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827090.655734:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564104832:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827090.655756:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827090.655767:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827090/real 1787827090] req@ffff976882aec3c0 x1874670564104832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827090.655777:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564104832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827090.655784:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827090.655787:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827090.655791:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564104832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827090.910984:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827090.910990:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827090.911064:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827091.678996:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827091.934930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827091.934931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827091.934957:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827092.702906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827092.958922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827092.958922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827092.958976:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827093.726930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827093.982921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827093.982999:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827093.983013:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827094.750978:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827095.006912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827095.006912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827095.008105:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827095.774922:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827095.774932:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827095.774937:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827095.774947:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2035 00000100:00080000:0.0:1787827095.774958:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827095.774976:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827095.775021:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564104960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827095.775029:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827095.775053:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827095.775059:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827095.775061:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827095.775066:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2035 00000100:00080000:0.0:1787827095.775070:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827095.775076:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827095.775087:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827095.775102:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564105088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827095.775108:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827095.775111:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827095.775112:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827095.775115:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2035 00000100:00080000:0.0:1787827095.775118:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827095.775120:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827095.775123:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827095.775130:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564105216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827095.775145:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827095.775152:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827095.775160:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827095.775164:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2035 00000100:00080000:0.0:1787827095.775167:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827095.775171:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827095.775176:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827095.775187:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564105344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827095.775196:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827095.775199:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827095.775201:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827095.775204:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2035 00000100:00080000:0.0:1787827095.775207:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827095.775210:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827095.775214:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827095.775222:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564105472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827095.775332:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564104960:192.168.203.139@tcp:250:kworker.0 00000800:00000100:2.0:1787827095.775394:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827095.775438:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827095/real 1787827095] req@ffff976882aed680 x1874670564104960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827095.775456:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564104960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827095.775468:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827095.775472:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827095.775475:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564104960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827095.775483:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564105088:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827095.775506:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827095.775519:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827095/real 1787827095] req@ffff976882aec3c0 x1874670564105088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827095.775553:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564105088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827095.775561:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827095.775565:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827095.775569:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564105088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827095.775592:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564105216:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827095.775618:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827095.775630:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827095/real 1787827095] req@ffff976882aeed00 x1874670564105216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827095.775642:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564105216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827095.775650:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827095.775653:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827095.775657:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564105216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827095.775663:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564105344:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827095.775683:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827095.775694:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827095/real 1787827095] req@ffff976882aee580 x1874670564105344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827095.775706:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564105344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827095.775714:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827095.775717:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827095.775721:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564105344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827095.775727:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564105472:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827095.775753:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827095.775766:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827095/real 1787827095] req@ffff976882aee940 x1874670564105472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827095.775776:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564105472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827095.775784:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827095.775788:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827095.775791:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564105472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827096.032115:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827096.033610:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827096.034098:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827096.798949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827097.054918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827097.058854:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827097.058939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827097.823004:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827098.078937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827098.079011:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827098.079113:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827098.846940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827099.102906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827099.102941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827099.102944:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827099.870944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827100.126927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827100.126933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827100.127034:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827100.895017:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827100.895026:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827100.895030:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827100.895039:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2040 00000100:00080000:1.0:1787827100.895051:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827100.895058:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827100.895092:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564105600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827100.895121:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827100.895124:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827100.895126:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827100.895130:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2040 00000100:00080000:1.0:1787827100.895134:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827100.895137:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827100.895146:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827100.895158:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564105728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827100.895178:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827100.895181:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827100.895183:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827100.895186:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2040 00000100:00080000:1.0:1787827100.895190:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827100.895193:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827100.895198:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827100.895209:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564105856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827100.895217:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827100.895220:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827100.895221:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827100.895224:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2040 00000100:00080000:1.0:1787827100.895227:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827100.895231:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827100.895235:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827100.895245:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564105984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827100.895253:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827100.895256:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827100.895257:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827100.895267:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2040 00000100:00080000:1.0:1787827100.895273:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827100.895275:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827100.895279:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827100.895288:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564106112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827100.895367:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827100.895477:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564105600:192.168.203.139@tcp:250:kworker.0 00000800:00000100:2.0:1787827100.895569:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827100.895617:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827100/real 1787827100] req@ffff976889eff480 x1874670564105600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827100.895634:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564105600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827100.895644:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827100.895648:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827100.895652:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564105600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827100.895658:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564105728:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827100.895684:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827100.895696:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827100/real 1787827100] req@ffff976889efc780 x1874670564105728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827100.895708:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564105728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827100.895716:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827100.895719:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827100.895722:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564105728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827100.895728:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564105856:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827100.895747:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827100.895757:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827100/real 1787827100] req@ffff976889efe1c0 x1874670564105856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827100.895769:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564105856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827100.895777:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827100.895780:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827100.895783:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564105856:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827100.895788:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564105984:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827100.895807:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827100.895817:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827100/real 1787827100] req@ffff976889efe580 x1874670564105984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827100.895861:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564105984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827100.895869:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827100.895871:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827100.895875:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564105984:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827100.895881:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564106112:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827100.895900:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827100.895910:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827100/real 1787827100] req@ffff976889efcb40 x1874670564106112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827100.895928:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564106112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827100.895948:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827100.895952:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827100.895955:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564106112:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827101.150919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827101.150942:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827101.151058:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827101.918935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827102.174916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827102.174916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827102.175047:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827102.942973:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827103.198963:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827103.198963:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827103.198979:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827103.966933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827104.222934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827104.222946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827104.223047:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827104.990905:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827105.246942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827105.246950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827105.247009:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827106.014942:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827106.014943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827106.014953:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827106.014958:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827106.014967:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2045 00000100:00080000:3.0:1787827106.014977:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827106.014985:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827106.015022:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564106240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827106.015048:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827106.015051:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827106.015053:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827106.015057:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2045 00000100:00080000:3.0:1787827106.015075:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827106.015080:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827106.015089:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827106.015095:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564106240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827106.015105:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564106368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827106.015117:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827106.015120:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827106.015122:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827106.015126:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2045 00000100:00080000:3.0:1787827106.015130:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827106.015133:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827106.015139:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827106.015150:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564106496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827106.015161:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787827106.015162:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827106.015167:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827106.015169:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827106.015175:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2045 00000100:00080000:3.0:1787827106.015179:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827106.015184:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827106.015189:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827106.015202:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564106624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827106.015211:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827106/real 1787827106] req@ffff97688cb5d2c0 x1874670564106240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827106.015214:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827106.015221:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827106.015225:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827106.015233:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2045 00000100:00080000:3.0:1787827106.015238:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827106.015241:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564106240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827106.015245:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827106.015254:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827106.015258:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827106.015264:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827106.015269:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564106752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827106.015270:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564106240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827106.015298:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564106368:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827106.015322:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827106.015335:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827106/real 1787827106] req@ffff97688cb5cf00 x1874670564106368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827106.015346:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564106368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827106.015354:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827106.015357:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827106.015360:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564106368:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827106.015365:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564106496:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827106.015381:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827106.015390:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827106/real 1787827106] req@ffff97688cb5f840 x1874670564106496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827106.015400:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564106496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827106.015413:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827106.015416:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827106.015418:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564106496:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827106.015423:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564106624:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827106.015438:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827106.015447:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827106/real 1787827106] req@ffff97688cb5c3c0 x1874670564106624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827106.015473:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564106624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827106.015480:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827106.015483:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827106.015486:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564106624:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827106.015491:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564106752:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827106.015510:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827106.015521:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827106/real 1787827106] req@ffff97688cb5e1c0 x1874670564106752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827106.015531:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564106752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827106.015538:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827106.015542:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827106.015545:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564106752:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827106.270964:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827106.271064:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827106.271084:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827107.038939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827107.294994:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827107.294994:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827107.295102:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827108.062938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827108.318968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827108.318982:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827108.319068:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827109.086929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827109.342953:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827109.343087:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827109.343270:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827110.110957:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827110.366946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827110.366992:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827110.367664:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827111.135219:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827111.136342:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827111.136349:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827111.136354:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827111.136365:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2051 00000100:00080000:1.0:1787827111.136378:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827111.136385:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827111.136432:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564106880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827111.136457:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827111.136461:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827111.136464:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827111.136470:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2051 00000100:00080000:1.0:1787827111.136474:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827111.136479:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827111.136489:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564106880:192.168.203.139@tcp:250:kworker.0 00000008:00100000:1.0:1787827111.136498:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827111.136521:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564107008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827111.136532:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827111.136535:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827111.136538:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827111.136543:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2051 00000100:00080000:1.0:1787827111.136548:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827111.136552:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827111.136559:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827111.136571:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564107136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827111.136581:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827111.136583:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827111.136585:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827111.136588:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2051 00000100:00080000:1.0:1787827111.136591:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827111.136595:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827111.136599:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827111.136610:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564107008:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827111.136611:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564107264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827111.136624:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827111.136626:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827111.136628:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827111.136630:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2051 00000100:00080000:1.0:1787827111.136633:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827111.136635:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827111.136638:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827111.136646:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564107392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827111.136662:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564107136:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827111.136691:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564107264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827111.136718:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564107392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827111.390954:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827111.391062:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827111.391872:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827112.158937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827112.414974:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827112.414978:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827112.415018:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827113.183214:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827113.438919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827113.438919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827113.438950:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827114.206898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:0.0:1787827114.207067:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:0.0:1787827114.207105:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827114.207120:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787827114.207157:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787827114.207166:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787827114.207173:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787827114.207180:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:3.0:1787827114.207259:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827111/real 1787827114] req@ffff976889efc780 x1874670564107264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827114.207279:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564107264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827114.207290:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827114.207303:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827114.207307:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564107264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827114.207314:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827111/real 1787827114] req@ffff976889eff480 x1874670564107392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827114.207325:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564107392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827114.207333:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827114.207336:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827114.207338:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564107392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827114.207343:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827111/real 1787827114] req@ffff976889efe580 x1874670564107008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827114.207354:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564107008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827114.207361:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827114.207365:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827114.207368:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564107008:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827114.207374:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827111/real 1787827114] req@ffff976889efe1c0 x1874670564107136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827114.207383:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564107136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827114.207391:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827114.207394:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827114.207407:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564107136:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827114.207413:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827111/real 1787827114] req@ffff976889efcb40 x1874670564106880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827114.207426:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564106880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827114.207434:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827114.207436:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827114.207439:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564106880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827114.462926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827114.462926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827114.463095:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827115.230943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827115.486933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827115.486935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827115.486962:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827116.254968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827116.257578:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827116.257588:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827116.257592:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827116.257601:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2056 00000100:00080000:3.0:1787827116.257614:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827116.257621:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827116.257660:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564107520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827116.257690:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827116.257693:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827116.257695:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827116.257699:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2056 00000100:00080000:3.0:1787827116.257703:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827116.257707:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827116.257717:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827116.257737:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564107520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827116.257742:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564107648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827116.257754:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827116.257760:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827116.257762:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827116.257765:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2056 00000100:00080000:3.0:1787827116.257770:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827116.257774:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827116.257779:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827116.257790:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564107776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827116.257800:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827116.257803:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827116.257805:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787827116.257810:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827116.257813:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2056 00000100:00080000:3.0:1787827116.257818:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827116.257839:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827116.257845:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827116.257857:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564107904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827116.257867:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827116.257890:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827116/real 1787827116] req@ffff97688cb5c780 x1874670564107520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827116.257910:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564107520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827116.257920:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827116.257924:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827116.257928:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564107520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827116.257952:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564107648:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827116.257975:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827116.257987:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827116/real 1787827116] req@ffff97688cb5cb40 x1874670564107648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827116.257999:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564107648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827116.258008:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827116.258011:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827116.258014:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564107648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827116.258021:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564107776:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787827116.258024:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827116.258029:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827116.258034:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2056 00000100:00080000:3.0:1787827116.258040:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827116.258045:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827116.258048:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827116.258056:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827116.258061:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827116/real 1787827116] req@ffff97688cb5c000 x1874670564107776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827116.258073:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564108032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827116.258075:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564107776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827116.258100:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827116.258104:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827116.258108:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564107776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827116.258114:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564107904:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827116.258133:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827116.258144:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827116/real 1787827116] req@ffff97688cb5d680 x1874670564107904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827116.258161:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564107904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827116.258167:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827116.258168:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827116.258171:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564107904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827116.258195:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564108032:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827116.258213:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827116.258223:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827116/real 1787827116] req@ffff97688cb5f0c0 x1874670564108032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827116.258234:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564108032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827116.258241:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827116.258244:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827116.258247:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564108032:192.168.203.139@tcp:8:kworker.0 00010000:00010000:1.0:1787827116.511027:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827116.511027:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827116.512851:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827117.278984:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827117.534943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827117.534972:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827117.535803:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827118.303592:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827118.558937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827118.558937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827118.559043:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827119.327006:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827119.582970:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827119.583026:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827119.583341:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827120.350898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827120.606929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827120.606935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827120.607401:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827121.374910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827121.375088:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827121.375097:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827121.375103:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827121.375115:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2061 00000100:00080000:3.0:1787827121.375129:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827121.375138:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827121.375188:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564108160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827121.375870:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827121.375876:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827121.375878:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827121.375885:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2061 00000100:00080000:3.0:1787827121.375892:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827121.375897:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827121.375909:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827121.375930:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564108288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827121.375941:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827121.375945:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827121.375947:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827121.375952:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2061 00000100:00080000:3.0:1787827121.375955:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827121.375959:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827121.375963:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827121.375973:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564108416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827121.375981:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827121.375984:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827121.375985:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827121.375989:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2061 00000100:00080000:3.0:1787827121.375992:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827121.375996:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827121.376001:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827121.376011:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564108544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827121.376020:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827121.376023:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827121.376025:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827121.376028:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2061 00000100:00080000:3.0:1787827121.376031:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827121.376035:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827121.376039:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827121.376049:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564108672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827121.377689:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564108160:192.168.203.139@tcp:250:kworker.0 00000800:00000100:2.0:1787827121.377763:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827121.377820:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827121/real 1787827121] req@ffff97688cb5de00 x1874670564108160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827121.377861:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564108160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827121.377872:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827121.377876:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827121.377880:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564108160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827121.377887:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564108288:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827121.377911:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827121.377924:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827121/real 1787827121] req@ffff97688cb5f0c0 x1874670564108288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827121.377935:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564108288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827121.377942:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827121.377946:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827121.377949:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564108288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827121.377955:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564108416:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827121.377975:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827121.377985:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827121/real 1787827121] req@ffff97688cb5d680 x1874670564108416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827121.377996:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564108416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827121.378008:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827121.378010:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827121.378013:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564108416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827121.378020:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564108544:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827121.378037:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827121.378049:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827121/real 1787827121] req@ffff97688cb5c000 x1874670564108544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827121.378059:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564108544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827121.378066:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827121.378069:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827121.378072:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564108544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827121.378078:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564108672:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827121.378095:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827121.378105:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827121/real 1787827121] req@ffff97688cb5cb40 x1874670564108672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827121.378117:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564108672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827121.378124:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827121.378127:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827121.378130:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564108672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827121.631028:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827121.631193:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827121.631215:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827122.398949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787827122.654957:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827122.654962:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827122.655358:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827123.422994:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827123.678957:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827123.679321:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827123.680478:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827124.447145:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827124.702938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827124.702965:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827124.703031:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827125.470979:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827125.726959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827125.726968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827125.726977:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827126.494941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827126.494966:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827126.494970:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827126.494974:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827126.494983:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2066 00000100:00080000:1.0:1787827126.494993:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827126.495001:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827126.495038:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564108800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827126.495076:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827126.495080:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827126.495081:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827126.495086:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2066 00000100:00080000:1.0:1787827126.495089:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827126.495094:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827126.495104:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827126.495116:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564108928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827126.495122:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827126.495125:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827126.495126:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827126.495129:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2066 00000100:00080000:1.0:1787827126.495131:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827126.495135:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827126.495138:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827126.495144:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564109056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827126.495156:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564108800:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827126.495162:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827126.495167:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827126.495168:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827126.495174:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2066 00000100:00080000:1.0:1787827126.495178:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827126.495183:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827126.495188:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827126.495202:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564109184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827126.495213:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827126.495216:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827126.495218:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827126.495222:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2066 00000100:00080000:1.0:1787827126.495226:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827126.495231:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:2.0:1787827126.495237:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787827126.495238:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827126.495257:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564109312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827126.495288:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827126/real 1787827126] req@ffff976889efc3c0 x1874670564108800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827126.495303:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564108800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827126.495324:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827126.495328:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827126.495331:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564108800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827126.495338:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564108928:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827126.495360:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827126.495374:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827126/real 1787827126] req@ffff976889eff0c0 x1874670564108928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827126.495386:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564108928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827126.495397:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827126.495401:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827126.495404:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564108928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827126.495429:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564109056:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827126.495445:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827126.495453:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827126/real 1787827126] req@ffff976889effc00 x1874670564109056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827126.495465:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564109056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827126.495472:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827126.495475:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827126.495479:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564109056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827126.495485:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564109184:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827126.495504:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827126.495512:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827126/real 1787827126] req@ffff976889efed00 x1874670564109184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827126.495519:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564109184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827126.495524:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827126.495526:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827126.495528:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564109184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827126.495532:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564109312:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827126.495543:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827126.495549:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827126/real 1787827126] req@ffff976889efe940 x1874670564109312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827126.495555:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564109312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827126.495560:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827126.495561:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827126.495570:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564109312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827126.750971:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827126.751025:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827126.751080:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827127.518977:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827127.775002:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827127.775043:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827127.775053:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827128.543164:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827128.798935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827128.798952:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827128.799025:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827129.566940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827129.822957:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827129.822960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827129.822998:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827130.590951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827130.846929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827130.846929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827130.847020:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827131.614938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827131.615072:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827131.615078:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827131.615081:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827131.615088:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2071 00000100:00080000:3.0:1787827131.615096:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827131.615102:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827131.615131:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564109440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827131.615153:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827131.615155:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827131.615156:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827131.615159:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2071 00000100:00080000:3.0:1787827131.615161:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827131.615163:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827131.615170:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827131.615178:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564109568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827131.615184:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827131.615188:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787827131.615190:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564109440:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827131.615190:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827131.615196:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2071 00000100:00080000:3.0:1787827131.615199:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827131.615202:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827131.615205:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827131.615216:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564109696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827131.615225:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827131.615228:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827131.615230:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827131.615235:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2071 00000100:00080000:3.0:1787827131.615239:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827131.615243:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:2.0:1787827131.615246:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:3.0:1787827131.615250:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827131.615285:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564109824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827131.615291:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827131/real 1787827131] req@ffff97688cb5c780 x1874670564109440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827131.615298:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827131.615304:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827131.615307:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827131.615312:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2071 00000100:00080000:3.0:1787827131.615315:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827131.615320:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827131.615321:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564109440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827131.615324:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827131.615343:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564109952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827131.615346:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827131.615356:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827131.615362:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564109440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827131.615386:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564109568:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827131.615407:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827131.615418:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827131/real 1787827131] req@ffff97688cb5e1c0 x1874670564109568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827131.615429:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564109568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827131.615438:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827131.615441:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827131.615444:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564109568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827131.615450:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564109696:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827131.615468:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827131.615477:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827131/real 1787827131] req@ffff97688cb5c3c0 x1874670564109696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827131.615484:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564109696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827131.615489:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827131.615491:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827131.615494:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564109696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827131.615500:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564109824:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827131.615517:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827131.615527:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827131/real 1787827131] req@ffff97688cb5f840 x1874670564109824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827131.615537:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564109824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827131.615545:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827131.615549:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827131.615551:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564109824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827131.615557:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564109952:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827131.615570:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827131.615578:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827131/real 1787827131] req@ffff97688cb5cf00 x1874670564109952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827131.615588:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564109952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827131.615596:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827131.615599:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827131.615602:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564109952:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827131.870905:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827131.870905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827131.870923:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827132.638931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827132.894929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827132.894929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827132.894988:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827133.662944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827133.918915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827133.918942:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827133.918965:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827134.687011:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827134.942927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787827134.942927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827134.943029:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827135.710954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787827135.966975:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827135.967329:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827135.967400:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00080000:3.0:1787827136.734981:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827136.734994:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827136.734999:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827136.735011:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2076 00000100:00080000:3.0:1787827136.735025:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827136.735033:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827136.735076:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564110080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827136.735105:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827136.735109:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827136.735111:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827136.735115:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2076 00000100:00080000:3.0:1787827136.735118:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827136.735122:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827136.735134:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827136.735151:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564110208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827136.735165:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564110080:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827136.735171:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827136.735176:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827136.735178:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827136.735182:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2076 00000100:00080000:3.0:1787827136.735186:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827136.735190:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827136.735194:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827136.735207:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564110336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827136.735216:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827136.735220:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827136.735222:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827136.735227:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2076 00000100:00080000:3.0:1787827136.735231:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827136.735235:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827136.735253:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1787827136.735259:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827136.735268:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564110464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827136.735279:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827136.735282:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827136.735284:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827136.735288:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2076 00000100:00080000:3.0:1787827136.735294:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827136.735299:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827136.735304:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827136.735319:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564110592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827136.735334:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827136/real 1787827136] req@ffff97688cb5d2c0 x1874670564110080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827136.735354:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827136.735359:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564110080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827136.735381:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827136.735387:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827136.735395:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564110080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827136.735426:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564110208:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827136.735448:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827136.735459:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827136/real 1787827136] req@ffff97688cb5cf00 x1874670564110208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827136.735470:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564110208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827136.735479:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827136.735482:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827136.735485:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564110208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827136.735491:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564110336:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827136.735505:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827136.735512:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827136/real 1787827136] req@ffff97688cb5f840 x1874670564110336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827136.735523:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564110336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827136.735531:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827136.735534:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827136.735537:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564110336:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827136.735542:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564110464:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827136.735556:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827136.735564:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827136/real 1787827136] req@ffff97688cb5c3c0 x1874670564110464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827136.735574:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564110464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827136.735582:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827136.735584:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827136.735587:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564110464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827136.735591:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564110592:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827136.735604:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827136.735611:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827136/real 1787827136] req@ffff97688cb5e1c0 x1874670564110592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827136.735621:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564110592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827136.735629:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827136.735631:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827136.735634:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564110592:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827136.990939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827136.990967:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827136.991039:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827137.758941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827138.014952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827138.015074:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827138.015921:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827138.782939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827139.038918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827139.038921:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827139.038971:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827139.806984:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827140.062945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827140.062947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827140.063244:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827140.831439:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827141.086938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827141.086938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827141.087043:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827141.854942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827141.854953:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827141.854961:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827141.854966:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827141.854985:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2081 00000100:00080000:1.0:1787827141.854998:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827141.855007:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827141.855059:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564110720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827141.855087:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827141.855089:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827141.855091:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827141.855094:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2081 00000100:00080000:1.0:1787827141.855097:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827141.855099:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827141.855108:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827141.855119:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564110848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827141.855132:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827141.855135:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827141.855136:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827141.855142:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2081 00000100:00100000:2.0:1787827141.855143:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564110720:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827141.855152:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827141.855158:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827141.855161:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827141.855171:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564110976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827141.855181:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827141.855185:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827141.855186:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827141.855191:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2081 00000100:00080000:1.0:1787827141.855196:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827141.855201:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827141.855217:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827141.855230:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564111104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827141.855241:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787827141.855251:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827141.855255:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827141.855257:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827141.855264:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2081 00000100:00080000:1.0:1787827141.855270:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827141.855276:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827141.855281:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827141.855289:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827141/real 1787827141] req@ffff976889efe940 x1874670564110720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827141.855301:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564111232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827141.855311:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564110720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827141.855325:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827141.855330:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827141.855334:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564110720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827141.855359:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564110848:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827141.855383:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827141.855394:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827141/real 1787827141] req@ffff976889efed00 x1874670564110848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827141.855404:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564110848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827141.855411:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827141.855415:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827141.855418:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564110848:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827141.855424:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564110976:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827141.855440:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827141.855448:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827141/real 1787827141] req@ffff976889effc00 x1874670564110976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827141.855459:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564110976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827141.855467:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827141.855470:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827141.855472:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564110976:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827141.855478:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564111104:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827141.855495:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827141.855504:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827141/real 1787827141] req@ffff976889eff0c0 x1874670564111104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827141.855514:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564111104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827141.855521:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827141.855524:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827141.855527:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564111104:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827141.855532:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564111232:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827141.855547:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827141.855555:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827141/real 1787827141] req@ffff976889efc3c0 x1874670564111232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827141.855567:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564111232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827141.855579:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827141.855582:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827141.855585:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564111232:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827142.110952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827142.111216:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827142.111267:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827142.878891:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827143.134945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827143.134947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827143.135115:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827143.902911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827144.158940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827144.158948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827144.158966:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827144.926903:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827145.182917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827145.182917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827145.183011:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827145.950971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827146.206951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827146.206962:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827146.206987:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827146.974944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827146.975006:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827146.975011:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827146.975018:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827146.975089:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2086 00000100:00080000:0.0:1787827146.975100:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827146.975107:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827146.975147:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564111360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827146.975191:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827146.975195:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827146.975197:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827146.975201:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2086 00000100:00080000:0.0:1787827146.975205:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827146.975208:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827146.975218:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827146.975235:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564111488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827146.975247:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827146.975251:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827146.975253:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827146.975258:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2086 00000100:00080000:0.0:1787827146.975263:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827146.975266:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827146.975272:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827146.975286:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564111616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827146.975295:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827146.975298:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827146.975301:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827146.975305:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2086 00000100:00080000:0.0:1787827146.975309:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827146.975312:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827146.975319:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827146.975329:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564111744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827146.975351:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827146.975355:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827146.975357:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827146.975360:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2086 00000100:00080000:0.0:1787827146.975364:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827146.975368:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827146.975372:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827146.975380:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564111360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827146.975387:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564111872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827146.975480:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827146.975524:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827146/real 1787827146] req@ffff976882aef0c0 x1874670564111360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827146.975542:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564111360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827146.975554:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827146.975558:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827146.975562:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564111360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827146.975571:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564111488:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827146.975592:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827146.975603:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827146/real 1787827146] req@ffff976882aee940 x1874670564111488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827146.975614:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564111488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827146.975635:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827146.975638:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827146.975642:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564111488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827146.975648:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564111616:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827146.975665:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827146.975674:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827146/real 1787827146] req@ffff976882aee580 x1874670564111616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827146.975684:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564111616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827146.975691:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827146.975694:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827146.975698:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564111616:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827146.975703:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564111744:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827146.975727:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827146.975737:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827146/real 1787827146] req@ffff976882aeed00 x1874670564111744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827146.975747:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564111744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827146.975755:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827146.975758:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827146.975761:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564111744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827146.975792:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564111872:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827146.975816:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827146.975863:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827146/real 1787827146] req@ffff976882aec3c0 x1874670564111872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827146.975878:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564111872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827146.975886:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827146.975890:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827146.975893:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564111872:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827147.230963:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827147.231024:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827147.231085:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827147.998979:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827148.254979:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827148.255012:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827148.255065:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827149.022944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827149.278962:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827149.278962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827149.279037:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827150.046985:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827150.302932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827150.302936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827150.302962:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827151.070995:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827151.326935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827151.327012:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827151.327044:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00080000:0.0:1787827152.094934:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827152.094943:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1787827152.094944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827152.094947:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827152.094956:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2091 00000100:00080000:0.0:1787827152.094965:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827152.094973:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827152.095016:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564112000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827152.095041:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827152.095045:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827152.095046:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827152.095058:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2091 00000100:00080000:0.0:1787827152.095063:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827152.095067:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827152.095077:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827152.095087:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564112000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827152.095091:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564112128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827152.095101:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827152.095103:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827152.095105:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827152.095107:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2091 00000100:00080000:0.0:1787827152.095110:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827152.095113:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827152.095116:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827152.095123:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564112256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827152.095146:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827152.095150:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827152.095152:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827152.095156:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2091 00000100:00080000:0.0:1787827152.095160:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827152.095166:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:1.0:1787827152.095167:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:0.0:1787827152.095183:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827152.095197:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564112384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827152.095207:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827152.095208:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827152/real 1787827152] req@ffff976882aed680 x1874670564112000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827152.095214:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827152.095217:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827152.095226:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2091 00000100:00080000:0.0:1787827152.095231:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827152.095234:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564112000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827152.095239:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827152.095247:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827152.095253:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787827152.095262:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564112512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827152.095262:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827152.095269:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564112000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827152.095292:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564112128:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827152.095322:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827152.095335:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827152/real 1787827152] req@ffff976882aec3c0 x1874670564112128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827152.095348:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564112128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827152.095354:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827152.095357:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827152.095360:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564112128:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827152.095365:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564112256:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827152.095382:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827152.095392:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827152/real 1787827152] req@ffff976882aeed00 x1874670564112256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827152.095404:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564112256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827152.095411:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827152.095414:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827152.095417:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564112256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827152.095422:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564112384:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827152.095449:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827152.095459:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827152/real 1787827152] req@ffff976882aee580 x1874670564112384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827152.095470:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564112384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827152.095478:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827152.095489:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827152.095492:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564112384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827152.095497:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564112512:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827152.095511:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827152.095517:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827152/real 1787827152] req@ffff976882aee940 x1874670564112512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827152.095524:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564112512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827152.095528:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827152.095530:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827152.095532:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564112512:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827152.350928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827152.350929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827152.351264:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827153.118958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827153.374980:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827153.375056:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827153.376070:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827154.143103:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827154.398920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827154.398921:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827154.399219:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827155.166950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827155.422943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827155.422943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827155.423246:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827156.190963:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827156.446931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827156.446955:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827156.447036:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827157.215084:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827157.215132:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827157.215137:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827157.215140:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827157.215147:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2097 00000100:00080000:1.0:1787827157.215157:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827157.215164:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827157.215199:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564112640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827157.215268:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564112640:192.168.203.139@tcp:250:kworker.0 00000800:00000100:3.0:1787827157.215328:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787827157.215401:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827157.215406:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827157.215408:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827157.215412:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2097 00000100:00080000:1.0:1787827157.215417:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827157.215421:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827157.215430:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827157.215454:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564112768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827157.215704:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827157/real 1787827157] req@ffff976889efcb40 x1874670564112640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827157.215722:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564112640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827157.215731:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827157.215735:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827157.215738:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564112640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827157.215760:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564112768:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827157.215799:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827157.215810:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827157/real 1787827157] req@ffff976889efe1c0 x1874670564112768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827157.215858:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564112768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827157.215867:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827157.215871:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827157.215874:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564112768:192.168.203.139@tcp:38:kworker.0 00000100:00080000:1.0:1787827157.216259:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827157.216264:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827157.216266:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827157.216270:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2097 00000100:00080000:1.0:1787827157.216275:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827157.216279:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827157.216287:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827157.216302:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564112896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827157.216332:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827157.216336:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827157.216338:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827157.216342:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2097 00000100:00080000:1.0:1787827157.216345:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827157.216349:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827157.216353:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827157.216364:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564113024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827157.216373:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827157.216376:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827157.216378:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827157.216381:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2097 00000100:00080000:1.0:1787827157.216384:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827157.216387:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827157.216391:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827157.216401:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564113152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827157.216511:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564112896:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827157.216542:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827157.216557:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827157/real 1787827157] req@ffff976889efe580 x1874670564112896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827157.216569:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564112896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827157.216578:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827157.216581:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827157.216585:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564112896:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827157.216591:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564113024:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827157.216611:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827157.216623:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827157/real 1787827157] req@ffff976889eff480 x1874670564113024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827157.216641:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564113024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827157.216649:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827157.216652:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827157.216656:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564113024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827157.216661:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564113152:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827157.216680:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827157.216690:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827157/real 1787827157] req@ffff976889efc780 x1874670564113152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827157.216702:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564113152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827157.216709:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827157.216712:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827157.216715:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564113152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827157.470936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827157.471025:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827157.471035:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827158.240196:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827158.494940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827158.494959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827158.494996:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827159.262985:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827159.518901:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827159.518959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827159.518971:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827160.287222:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827160.542893:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827160.542910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827160.542985:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827161.310926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827161.566893:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827161.566913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827161.566964:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827162.334947:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827162.334957:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827162.334961:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827162.334981:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2102 00000100:00080000:3.0:1787827162.334994:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827162.334996:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827162.335002:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827162.335040:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564113280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827162.335085:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827162.335089:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827162.335091:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827162.335096:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2102 00000100:00080000:3.0:1787827162.335101:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827162.335106:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827162.335120:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827162.335124:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564113280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827162.335147:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564113408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827162.335158:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827162.335161:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827162.335163:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827162.335166:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2102 00000100:00080000:3.0:1787827162.335170:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827162.335173:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827162.335178:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827162.335188:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564113536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827162.335197:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787827162.335201:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827162.335202:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827162.335206:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827162.335211:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2102 00000100:00080000:3.0:1787827162.335216:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827162.335220:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827162.335235:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827162.335240:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827162/real 1787827162] req@ffff97688cb5c780 x1874670564113280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827162.335250:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564113664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827162.335261:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564113280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827162.335262:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827162.335272:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827162.335275:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827162.335280:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827162.335281:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2102 00000100:00080000:3.0:1787827162.335288:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827162.335289:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:3.0:1787827162.335295:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827162.335298:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564113280:192.168.203.139@tcp:250:kworker.0 00000008:00100000:3.0:1787827162.335302:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827162.335321:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564113792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827162.335324:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564113408:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827162.335354:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827162.335373:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827162/real 1787827162] req@ffff97688cb5cb40 x1874670564113408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827162.335387:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564113408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827162.335396:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827162.335399:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827162.335402:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564113408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827162.335407:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564113536:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827162.335425:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827162.335443:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827162/real 1787827162] req@ffff97688cb5c000 x1874670564113536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827162.335463:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564113536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827162.335476:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827162.335480:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827162.335482:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564113536:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827162.335488:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564113664:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827162.335506:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827162.335517:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827162/real 1787827162] req@ffff97688cb5d680 x1874670564113664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827162.335528:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564113664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827162.335535:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827162.335538:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827162.335541:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564113664:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827162.335557:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564113792:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827162.335574:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827162.335585:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827162/real 1787827162] req@ffff97688cb5f0c0 x1874670564113792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827162.335594:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564113792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827162.335608:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827162.335611:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827162.335614:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564113792:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827162.590953:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827162.591239:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827162.591380:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827163.359001:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827163.614932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827163.615023:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827163.615160:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827164.382945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827164.638914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827164.638945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827164.638984:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827165.406991:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827165.662941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827165.662948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827165.663013:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827166.430918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827166.686979:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827166.686987:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827166.687071:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827167.454934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827167.454970:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827167.454981:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827167.454987:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827167.455000:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2107 00000100:00080000:3.0:1787827167.455018:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827167.455039:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827167.455087:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564113920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827167.455118:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827167.455124:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827167.455126:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827167.455130:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2107 00000100:00080000:3.0:1787827167.455134:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827167.455137:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827167.455148:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827167.455164:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564114048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827167.455170:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564113920:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827167.455179:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827167.455183:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827167.455184:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827167.455188:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2107 00000100:00080000:3.0:1787827167.455191:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827167.455193:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827167.455198:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827167.455208:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564114176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827167.455218:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827167.455221:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827167.455223:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827167.455227:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2107 00000100:00080000:3.0:1787827167.455231:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827167.455234:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827167.455240:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827167.455254:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564114304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827167.455255:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827167.455266:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827167.455270:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827167.455271:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827167.455276:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2107 00000100:00080000:3.0:1787827167.455280:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827167.455284:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827167.455288:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827167.455299:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564114432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827167.455307:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827167/real 1787827167] req@ffff97688cb5de00 x1874670564113920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827167.455327:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564113920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827167.455338:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827167.455342:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827167.455346:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564113920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827167.455373:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564114048:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827167.455395:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827167.455405:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827167/real 1787827167] req@ffff97688cb5f0c0 x1874670564114048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827167.455413:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564114048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827167.455419:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827167.455422:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827167.455424:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564114048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827167.455428:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564114176:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827167.455439:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827167.455444:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827167/real 1787827167] req@ffff97688cb5d680 x1874670564114176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827167.455451:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564114176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827167.455456:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827167.455458:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827167.455459:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564114176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827167.455462:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564114304:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827167.455472:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827167.455477:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827167/real 1787827167] req@ffff97688cb5c000 x1874670564114304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827167.455484:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564114304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827167.455488:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827167.455489:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827167.455492:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564114304:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827167.455495:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564114432:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827167.455505:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827167.455510:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827167/real 1787827167] req@ffff97688cb5cb40 x1874670564114432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827167.455516:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564114432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827167.455521:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827167.455523:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827167.455524:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564114432:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827167.710926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827167.710959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827167.711046:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827168.478891:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827168.734906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827168.734906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827168.735003:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827169.502944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827169.758922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827169.758922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827169.758955:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827170.526918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827170.782935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827170.782942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827170.783123:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827171.550938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827171.806972:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827171.807073:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827171.807078:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827172.574939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827172.574978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827172.574988:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827172.575000:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827172.575012:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2112 00000100:00080000:1.0:1787827172.575027:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827172.575035:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827172.575078:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564114560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827172.575106:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827172.575109:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827172.575110:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827172.575113:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2112 00000100:00080000:1.0:1787827172.575116:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827172.575119:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827172.575129:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827172.575137:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564114688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827172.575145:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827172.575150:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827172.575152:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827172.575152:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564114560:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827172.575160:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2112 00000100:00080000:1.0:1787827172.575167:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827172.575171:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827172.575176:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827172.575185:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564114816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827172.575203:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827172.575206:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827172.575208:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827172.575213:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2112 00000100:00080000:1.0:1787827172.575218:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827172.575223:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827172.575230:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827172.575245:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564114944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827172.575249:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787827172.575258:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827172.575261:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827172.575262:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827172.575266:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2112 00000100:00080000:1.0:1787827172.575271:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827172.575274:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827172.575278:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827172.575284:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564115072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827172.575315:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827172/real 1787827172] req@ffff976889efc780 x1874670564114560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827172.575340:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564114560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827172.575350:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827172.575355:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827172.575359:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564114560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827172.575381:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564114688:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827172.575404:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827172.575415:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827172/real 1787827172] req@ffff976889eff480 x1874670564114688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827172.575427:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564114688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827172.575434:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827172.575437:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827172.575440:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564114688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827172.575447:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564114816:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827172.575464:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827172.575475:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827172/real 1787827172] req@ffff976889efe580 x1874670564114816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827172.575486:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564114816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827172.575494:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827172.575497:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827172.575500:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564114816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827172.575506:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564114944:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827172.575525:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827172.575535:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827172/real 1787827172] req@ffff976889efe1c0 x1874670564114944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827172.575552:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564114944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827172.575560:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827172.575563:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827172.575566:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564114944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827172.575572:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564115072:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827172.575590:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827172.575600:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827172/real 1787827172] req@ffff976889efcb40 x1874670564115072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827172.575610:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564115072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827172.575618:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827172.575621:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827172.575624:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564115072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827172.830893:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827172.830897:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827172.830986:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827173.599005:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827173.854939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827173.854948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827173.854975:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827174.622991:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827174.878930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827174.878930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827174.878993:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827175.646985:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827175.902989:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827175.902989:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827175.903056:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827176.670936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827176.926926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827176.926926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827176.926986:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827177.694952:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827177.694973:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827177.694977:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827177.694987:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2117 00000100:00080000:1.0:1787827177.694999:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827177.695006:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827177.695042:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564115200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827177.695073:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827177.695083:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787827177.695084:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827177.695088:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827177.695100:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2117 00000100:00080000:1.0:1787827177.695104:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827177.695108:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827177.695118:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827177.695129:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564115200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827177.695132:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564115328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827177.695145:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827177.695148:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827177.695150:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827177.695154:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2117 00000100:00080000:1.0:1787827177.695157:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827177.695161:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827177.695166:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827177.695176:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564115456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827177.695186:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827177.695189:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827177.695191:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827177.695195:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2117 00000100:00080000:1.0:1787827177.695199:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827177.695203:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827177.695208:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827177.695218:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564115584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827177.695227:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827177.695231:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787827177.695233:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564115328:192.168.203.139@tcp:38:kworker.0 00000100:00080000:1.0:1787827177.695234:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827177.695243:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2117 00000100:00080000:1.0:1787827177.695248:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827177.695253:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827177.695258:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827177.695265:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564115456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827177.695271:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564115712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827177.695282:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564115584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827177.695311:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564115712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827177.950919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827177.950923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827177.951004:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827178.719030:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827178.974961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827178.974963:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827178.975004:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827179.743029:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827179.998952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827179.998966:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827179.998986:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827180.767165:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:3.0:1787827180.767297:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:3.0:1787827180.767359:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787827180.767386:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787827180.767439:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787827180.767448:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787827180.767456:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:3.0:1787827180.767463:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:1.0:1787827180.767747:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827177/real 1787827180] req@ffff976889efe940 x1874670564115712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827180.767778:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564115712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827180.767789:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827180.767794:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827180.767799:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564115712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827180.767807:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827177/real 1787827180] req@ffff976889eff0c0 x1874670564115328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827180.767818:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564115328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827180.767866:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827180.767871:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827180.767874:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564115328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827180.767880:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827177/real 1787827180] req@ffff976889effc00 x1874670564115456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827180.767892:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564115456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827180.767899:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827180.767903:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827180.767906:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564115456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827180.767913:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827177/real 1787827180] req@ffff976889efed00 x1874670564115584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827180.767924:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564115584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827180.767932:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827180.767948:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827180.767952:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564115584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827180.767959:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827177/real 1787827180] req@ffff976889efc3c0 x1874670564115200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827180.767971:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564115200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827180.767978:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827180.767981:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827180.767984:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564115200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827181.022892:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827181.022946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827181.022966:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827181.790971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827182.046890:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827182.046913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827182.046977:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827182.814981:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827182.815004:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827182.815010:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827182.815015:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827182.815024:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2122 00000100:00080000:1.0:1787827182.815033:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827182.815040:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827182.815071:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564115840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827182.815104:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827182.815108:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827182.815110:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827182.815115:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2122 00000100:00080000:1.0:1787827182.815119:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827182.815124:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827182.815126:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564115840:192.168.203.139@tcp:250:kworker.0 00000008:00100000:1.0:1787827182.815140:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827182.815155:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564115968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827182.815163:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827182.815166:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827182.815167:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827182.815170:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2122 00000100:00080000:1.0:1787827182.815177:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827182.815181:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827182.815186:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827182.815189:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827182.815200:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564116096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827182.815209:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827182.815212:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827182.815214:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827182.815219:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2122 00000100:00080000:1.0:1787827182.815222:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827182.815226:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827182.815231:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827182.815235:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827182/real 1787827182] req@ffff976889efc3c0 x1874670564115840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827182.815245:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564116224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827182.815255:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564115840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827182.815257:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827182.815264:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827182.815267:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827182.815272:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2122 00000100:00080000:1.0:1787827182.815278:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827182.815283:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827182.815284:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:2.0:1787827182.815291:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000008:00100000:1.0:1787827182.815292:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827182.815297:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564115840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827182.815307:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564116352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827182.815325:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564115968:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827182.815352:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827182.815366:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827182/real 1787827182] req@ffff976889efed00 x1874670564115968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827182.815384:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564115968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827182.815393:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827182.815397:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827182.815401:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564115968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827182.815408:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564116096:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827182.815432:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827182.815445:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827182/real 1787827182] req@ffff976889effc00 x1874670564116096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827182.815457:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564116096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827182.815466:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827182.815469:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827182.815473:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564116096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827182.815480:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564116224:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827182.815501:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827182.815512:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827182/real 1787827182] req@ffff976889eff0c0 x1874670564116224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827182.815524:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564116224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827182.815537:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827182.815540:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827182.815543:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564116224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827182.815548:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564116352:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827182.815567:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827182.815577:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827182/real 1787827182] req@ffff976889efe940 x1874670564116352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827182.815588:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564116352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827182.815596:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827182.815599:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827182.815601:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564116352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827183.070931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827183.070934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827183.070993:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827183.838991:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827184.094896:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827184.094942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827184.094946:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827184.862927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827185.118921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827185.118921:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827185.119069:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827185.886920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827186.142904:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827186.142908:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827186.143024:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827186.910944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827187.166965:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827187.166972:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827187.167034:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827187.934881:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827187.934922:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827187.934927:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827187.934930:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827187.934936:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2127 00000100:00080000:3.0:1787827187.934943:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827187.934948:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827187.934977:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564116480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827187.934996:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827187.934998:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827187.934999:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827187.935001:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2127 00000100:00080000:3.0:1787827187.935003:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827187.935005:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827187.935011:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827187.935019:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564116608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827187.935031:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564116480:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827187.935031:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827187.935037:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827187.935039:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827187.935045:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2127 00000100:00080000:3.0:1787827187.935047:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827187.935049:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827187.935051:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827187.935058:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564116736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827187.935063:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827187.935065:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827187.935066:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827187.935069:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2127 00000100:00080000:3.0:1787827187.935071:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827187.935073:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827187.935076:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827187.935082:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564116864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827187.935087:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827187.935088:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827187.935089:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827187.935091:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2127 00000100:00080000:3.0:1787827187.935095:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827187.935096:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827187.935099:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827187.935103:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827187.935114:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564116992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827187.935131:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827187/real 1787827187] req@ffff97688cb5c780 x1874670564116480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827187.935144:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564116480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827187.935151:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827187.935153:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827187.935156:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564116480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827187.935178:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564116608:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827187.935198:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827187.935210:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827187/real 1787827187] req@ffff97688cb5e1c0 x1874670564116608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827187.935221:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564116608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827187.935228:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827187.935231:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827187.935233:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564116608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827187.935238:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564116736:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827187.935255:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827187.935265:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827187/real 1787827187] req@ffff97688cb5c3c0 x1874670564116736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827187.935282:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564116736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827187.935290:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827187.935293:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827187.935296:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564116736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827187.935302:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564116864:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827187.935320:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827187.935331:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827187/real 1787827187] req@ffff97688cb5f840 x1874670564116864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827187.935342:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564116864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827187.935349:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827187.935352:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827187.935355:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564116864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827187.935361:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564116992:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827187.935376:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827187.935384:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827187/real 1787827187] req@ffff97688cb5cf00 x1874670564116992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827187.935394:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564116992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827187.935406:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827187.935410:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827187.935413:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564116992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827188.190908:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827188.190910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827188.190983:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827188.958926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827189.214980:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827189.214981:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827189.215000:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827189.982945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827190.238924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827190.238928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827190.238985:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827191.006968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827191.262946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827191.262946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827191.263004:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827192.030939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827192.286903:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827192.286981:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827192.287024:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827193.054920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827193.054958:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827193.054966:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827193.054971:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827193.054984:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2132 00000100:00080000:0.0:1787827193.054996:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827193.055004:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827193.055048:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564117120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827193.055072:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827193.055075:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827193.055076:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827193.055079:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2132 00000100:00080000:0.0:1787827193.055082:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827193.055085:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827193.055095:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827193.055108:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564117248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827193.055118:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564117120:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787827193.055120:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827193.055133:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827193.055135:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827193.055140:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2132 00000100:00080000:0.0:1787827193.055142:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827193.055144:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827193.055149:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827193.055162:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564117376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827193.055177:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827193.055182:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827193.055183:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827193.055187:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2132 00000100:00080000:0.0:1787827193.055192:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827193.055196:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827193.055201:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1787827193.055207:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827193.055212:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564117504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827193.055221:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827193.055225:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827193.055228:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827193.055232:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2132 00000100:00080000:0.0:1787827193.055234:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827193.055237:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827193.055243:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827193.055251:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564117632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827193.055261:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827193/real 1787827193] req@ffff976882aef0c0 x1874670564117120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827193.055284:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564117120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827193.055296:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827193.055302:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827193.055307:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564117120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827193.055332:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564117248:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827193.055352:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827193.055362:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827193/real 1787827193] req@ffff976882aee940 x1874670564117248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827193.055371:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564117248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827193.055376:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827193.055379:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827193.055381:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564117248:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827193.055385:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564117376:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827193.055396:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827193.055402:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827193/real 1787827193] req@ffff976882aee580 x1874670564117376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827193.055409:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564117376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827193.055414:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827193.055416:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827193.055418:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564117376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827193.055421:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564117504:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827193.055431:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827193.055437:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827193/real 1787827193] req@ffff976882aeed00 x1874670564117504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827193.055444:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564117504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827193.055449:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827193.055451:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827193.055453:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564117504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827193.055457:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564117632:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827193.055466:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827193.055471:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827193/real 1787827193] req@ffff976882aec3c0 x1874670564117632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827193.055478:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564117632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827193.055483:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827193.055485:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827193.055487:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564117632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827193.310965:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827193.310972:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827193.311020:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827194.078962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827194.334906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827194.334934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827194.334967:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827195.102965:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827195.358967:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827195.358969:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827195.359038:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827196.126942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827196.382914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827196.382918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827196.382995:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827197.150943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827197.406933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827197.406934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827197.407033:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827198.174934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827198.174942:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827198.174950:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827198.174954:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827198.174965:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2138 00000100:00080000:3.0:1787827198.174976:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827198.174984:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827198.175021:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564117760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827198.175041:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827198.175044:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827198.175045:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827198.175048:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2138 00000100:00080000:3.0:1787827198.175051:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827198.175055:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827198.175063:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827198.175072:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564117760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827198.175077:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564117888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827198.175086:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827198.175090:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827198.175092:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827198.175094:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2138 00000100:00080000:3.0:1787827198.175096:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827198.175099:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827198.175102:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827198.175110:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564118016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827198.175116:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827198.175118:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827198.175120:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827198.175122:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2138 00000100:00080000:3.0:1787827198.175125:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827198.175128:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:2.0:1787827198.175133:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:3.0:1787827198.175135:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827198.175144:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564118144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827198.175151:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827198.175154:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827198.175155:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827198.175158:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2138 00000100:00080000:3.0:1787827198.175160:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827198.175162:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827198.175166:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827198.175174:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564118272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827198.175188:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827198/real 1787827198] req@ffff97688cb5d2c0 x1874670564117760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827198.175202:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564117760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827198.175210:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827198.175214:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827198.175220:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564117760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827198.175239:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564117888:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827198.175260:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827198.175273:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827198/real 1787827198] req@ffff97688cb5cf00 x1874670564117888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827198.175286:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564117888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827198.175295:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827198.175299:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827198.175303:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564117888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827198.175309:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564118016:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827198.175328:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827198.175336:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827198/real 1787827198] req@ffff97688cb5f840 x1874670564118016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827198.175349:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564118016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827198.175354:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827198.175356:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827198.175358:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564118016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827198.175362:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564118144:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827198.175373:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827198.175379:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827198/real 1787827198] req@ffff97688cb5c3c0 x1874670564118144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827198.175394:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564118144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827198.175400:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827198.175402:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827198.175404:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564118144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827198.175407:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564118272:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827198.175418:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827198.175424:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827198/real 1787827198] req@ffff97688cb5e1c0 x1874670564118272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827198.175431:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564118272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827198.175440:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827198.175442:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827198.175444:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564118272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827198.430955:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827198.430960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827198.431039:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827199.198920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827199.454989:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827199.455001:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827199.455057:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827200.222925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827200.478929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827200.478929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827200.479023:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827201.246896:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827201.502909:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827201.502910:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827201.502975:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827202.270933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827202.526923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827202.526923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827202.526993:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827203.294925:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827203.294928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827203.294937:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827203.294942:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827203.294953:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2143 00000100:00080000:3.0:1787827203.294964:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827203.294970:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827203.295003:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564118400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827203.295025:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827203.295028:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827203.295029:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827203.295032:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2143 00000100:00080000:3.0:1787827203.295035:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827203.295038:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827203.295046:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827203.295053:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564118400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827203.295057:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564118528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827203.295067:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827203.295069:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827203.295070:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827203.295073:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2143 00000100:00080000:3.0:1787827203.295077:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827203.295079:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827203.295082:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827203.295091:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564118656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827203.295098:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827203.295100:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827203.295102:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827203.295104:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2143 00000100:00080000:3.0:1787827203.295107:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827203.295109:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827203.295112:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827203.295119:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564118784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827203.295126:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827203.295132:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827203.295135:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827203.295137:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827203.295140:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2143 00000100:00080000:3.0:1787827203.295143:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827203.295146:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827203.295149:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827203.295157:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564118912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827203.295176:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827203/real 1787827203] req@ffff97688cb5c780 x1874670564118400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827203.295192:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564118400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827203.295201:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827203.295204:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827203.295207:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564118400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827203.295226:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564118528:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827203.295243:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827203.295252:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827203/real 1787827203] req@ffff97688cb5cb40 x1874670564118528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827203.295271:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564118528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827203.295278:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827203.295280:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827203.295282:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564118528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827203.295287:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564118656:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827203.295305:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827203.295316:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827203/real 1787827203] req@ffff97688cb5c000 x1874670564118656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827203.295329:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564118656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827203.295338:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827203.295341:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827203.295344:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564118656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827203.295351:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564118784:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827203.295369:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827203.295376:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827203/real 1787827203] req@ffff97688cb5d680 x1874670564118784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827203.295384:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564118784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827203.295389:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827203.295392:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827203.295394:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564118784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827203.295397:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564118912:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827203.295407:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827203.295412:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827203/real 1787827203] req@ffff97688cb5f0c0 x1874670564118912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827203.295419:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564118912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827203.295424:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827203.295426:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827203.295428:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564118912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827203.550906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827203.550906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827203.551007:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827204.318978:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827204.574959:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827204.574964:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827204.574987:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827205.342896:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827205.598904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827205.598923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827205.599001:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827206.366968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827206.622918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827206.622918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827206.622977:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827207.390879:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827207.646895:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827207.646903:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827207.647031:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827208.414911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827208.414919:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827208.414925:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827208.414928:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827208.414937:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2148 00000100:00080000:1.0:1787827208.414946:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827208.414951:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827208.414985:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564119040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827208.415007:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827208.415009:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827208.415011:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827208.415014:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2148 00000100:00080000:1.0:1787827208.415016:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827208.415019:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827208.415026:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827208.415036:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564119168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827208.415044:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827208.415047:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827208.415051:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827208.415051:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564119040:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827208.415057:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2148 00000100:00080000:1.0:1787827208.415063:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827208.415066:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827208.415071:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827208.415080:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564119296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827208.415094:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827208.415097:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827208.415098:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827208.415101:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2148 00000100:00080000:1.0:1787827208.415104:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827208.415107:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827208.415111:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827208.415122:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564119424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827208.415122:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787827208.415132:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827208.415135:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827208.415137:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827208.415140:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2148 00000100:00080000:1.0:1787827208.415143:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827208.415145:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827208.415148:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827208.415156:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564119552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827208.415162:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827208/real 1787827208] req@ffff976889efe940 x1874670564119040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827208.415178:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564119040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827208.415187:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827208.415190:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827208.415193:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564119040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827208.415210:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564119168:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827208.415225:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827208.415233:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827208/real 1787827208] req@ffff976889eff0c0 x1874670564119168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827208.415241:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564119168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827208.415247:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827208.415250:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827208.415252:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564119168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827208.415255:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564119296:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827208.415265:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827208.415271:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827208/real 1787827208] req@ffff976889effc00 x1874670564119296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827208.415282:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564119296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827208.415287:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827208.415289:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827208.415291:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564119296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827208.415295:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564119424:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827208.415305:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827208.415311:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827208/real 1787827208] req@ffff976889efed00 x1874670564119424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827208.415318:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564119424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827208.415323:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827208.415325:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827208.415326:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564119424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827208.415330:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564119552:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827208.415339:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827208.415344:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827208/real 1787827208] req@ffff976889efc3c0 x1874670564119552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827208.415351:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564119552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827208.415360:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827208.415362:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827208.415364:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564119552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827208.670925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827208.670929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827208.670981:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827209.438925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827209.694904:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827209.694926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827209.694944:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827210.462934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827210.718928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827210.718928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827210.718962:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827211.486932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827211.742891:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827211.742891:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827211.742957:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827212.510915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827212.766927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827212.766936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827212.766991:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827213.534936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827213.534947:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827213.534953:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827213.534957:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827213.534964:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2153 00000100:00080000:3.0:1787827213.534979:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827213.534985:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827213.535017:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564119680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827213.535036:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827213.535039:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827213.535040:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827213.535043:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2153 00000100:00080000:3.0:1787827213.535046:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827213.535049:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827213.535058:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827213.535065:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564119680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827213.535071:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564119808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827213.535080:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827213.535082:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827213.535084:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827213.535087:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2153 00000100:00080000:3.0:1787827213.535089:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827213.535091:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827213.535096:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827213.535104:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564119936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827213.535116:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827213.535119:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827213.535120:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787827213.535126:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827213.535126:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2153 00000100:00080000:3.0:1787827213.535131:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827213.535135:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827213.535139:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827213.535147:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564120064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827213.535154:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827213.535156:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827213.535157:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827213.535160:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2153 00000100:00080000:3.0:1787827213.535163:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827213.535166:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827213.535169:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827213.535175:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827213/real 1787827213] req@ffff97688cb5de00 x1874670564119680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827213.535180:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564120192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827213.535203:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564119680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827213.535214:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827213.535217:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827213.535220:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564119680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827213.535241:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564119808:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827213.535257:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827213.535266:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827213/real 1787827213] req@ffff97688cb5f0c0 x1874670564119808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827213.535274:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564119808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827213.535280:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827213.535283:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827213.535285:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564119808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827213.535288:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564119936:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827213.535299:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827213.535304:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827213/real 1787827213] req@ffff97688cb5d680 x1874670564119936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827213.535312:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564119936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827213.535317:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827213.535319:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827213.535321:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564119936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827213.535324:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564120064:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827213.535341:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827213.535347:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827213/real 1787827213] req@ffff97688cb5c000 x1874670564120064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827213.535360:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564120064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827213.535365:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827213.535367:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827213.535368:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564120064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827213.535372:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564120192:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827213.535383:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827213.535388:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827213/real 1787827213] req@ffff97688cb5cb40 x1874670564120192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827213.535395:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564120192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827213.535400:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827213.535402:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827213.535404:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564120192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827213.790920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827213.790920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827213.790978:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827214.558951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827214.814965:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827214.815003:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827214.815026:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827215.582962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827215.838959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827215.838959:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827215.839027:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827216.606948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827216.862942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827216.862943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827216.862999:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827217.630987:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827217.886937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827217.886989:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827217.887022:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827218.654982:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827218.655059:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827218.655066:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827218.655071:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827218.655082:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2158 00000100:00080000:2.0:1787827218.655095:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827218.655103:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827218.655153:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564120320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827218.655188:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827218.655193:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827218.655196:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827218.655202:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2158 00000100:00080000:2.0:1787827218.655210:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827218.655217:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827218.655228:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827218.655252:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564120320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827218.655253:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564120448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827218.655280:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827218.655285:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827218.655286:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827218.655292:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2158 00000100:00080000:2.0:1787827218.655296:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827218.655300:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827218.655306:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827218.655319:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564120576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827218.655330:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827218.655334:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827218.655336:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827218.655341:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2158 00000100:00080000:2.0:1787827218.655357:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787827218.655368:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787827218.655419:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827218.655426:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827218.655444:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827218/real 1787827218] req@ffff976889cf8f00 x1874670564120320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827218.655447:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564120704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827218.655528:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827218.655532:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827218.655535:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827218.655542:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564120320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827218.655544:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2158 00000100:00080000:2.0:1787827218.655556:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827218.655566:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:0.0:1787827218.655569:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787827218.655573:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787827218.655580:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827218.655588:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564120320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827218.655598:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564120832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827218.655637:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564120448:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827218.655688:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827218.655717:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827218/real 1787827218] req@ffff976889cfb0c0 x1874670564120448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827218.655735:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564120448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827218.655752:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827218.655758:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827218.655765:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564120448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827218.655776:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564120576:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827218.655804:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827218.655818:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827218/real 1787827218] req@ffff976889cf8000 x1874670564120576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827218.655867:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564120576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827218.655879:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827218.655884:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827218.655889:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564120576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827218.655897:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564120704:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827218.655921:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827218.655933:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827218/real 1787827218] req@ffff976889cfbc00 x1874670564120704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827218.655947:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564120704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827218.655956:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827218.655959:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827218.655962:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564120704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827218.655969:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564120832:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827218.655991:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827218.656003:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827218/real 1787827218] req@ffff976889cfa940 x1874670564120832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827218.656016:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564120832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827218.656025:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827218.656029:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827218.656032:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564120832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827218.910924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827218.910927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827218.911018:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827219.678941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787827219.934927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827219.934936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827219.935002:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827220.702899:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827220.958921:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827220.958930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827220.958981:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827221.726930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827221.982930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827221.982930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827221.982986:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827222.750913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827223.006903:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827223.006903:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827223.007068:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827223.774941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827223.774945:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827223.774954:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827223.774959:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827223.774969:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2163 00000100:00080000:0.0:1787827223.774980:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827223.774990:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827223.775042:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564120960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827223.775073:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827223.775077:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827223.775078:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827223.775083:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2163 00000100:00080000:0.0:1787827223.775087:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827223.775093:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827223.775103:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827223.775108:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564120960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827223.775123:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564121088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827223.775134:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827223.775137:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827223.775138:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827223.775141:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2163 00000100:00080000:0.0:1787827223.775144:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827223.775147:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827223.775151:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827223.775159:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564121216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827223.775166:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827223.775168:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827223.775169:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827223.775172:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2163 00000100:00080000:0.0:1787827223.775175:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827223.775181:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787827223.775182:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827223.775188:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827223.775196:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564121344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827223.775203:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827223.775206:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827223.775207:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827223.775210:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2163 00000100:00080000:0.0:1787827223.775212:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827223.775214:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827223.775218:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827223.775230:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564121472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827223.775230:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827223/real 1787827223] req@ffff976882aed680 x1874670564120960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827223.775271:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564120960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827223.775282:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827223.775286:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827223.775289:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564120960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827223.775309:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564121088:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827223.775327:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827223.775337:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827223/real 1787827223] req@ffff976882aec3c0 x1874670564121088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827223.775345:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564121088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827223.775351:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827223.775353:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827223.775356:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564121088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827223.775360:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564121216:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827223.775374:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827223.775384:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827223/real 1787827223] req@ffff976882aeed00 x1874670564121216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827223.775395:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564121216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827223.775404:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827223.775407:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827223.775412:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564121216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827223.775418:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564121344:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827223.775437:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827223.775449:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827223/real 1787827223] req@ffff976882aee580 x1874670564121344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827223.775460:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564121344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827223.775469:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827223.775472:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827223.775476:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564121344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827223.775482:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564121472:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827223.775502:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827223.775512:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827223/real 1787827223] req@ffff976882aee940 x1874670564121472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827223.775520:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564121472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827223.775525:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827223.775527:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827223.775529:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564121472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827224.030902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827224.030964:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827224.030986:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827224.798915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827225.054943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827225.054948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827225.055007:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827225.822954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827226.078908:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827226.078908:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827226.078913:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827226.846934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827227.102897:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827227.102897:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827227.102935:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827227.870931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827228.126897:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827228.126904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827228.126928:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827228.894920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827228.894941:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827228.894952:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827228.894957:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827228.894970:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2168 00000100:00080000:0.0:1787827228.894985:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827228.894994:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827228.895041:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564121600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827228.895076:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827228.895080:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827228.895082:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827228.895086:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2168 00000100:00080000:0.0:1787827228.895090:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827228.895095:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827228.895107:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827228.895132:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564121600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827228.895133:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564121728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827228.895160:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827228.895165:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827228.895166:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827228.895171:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2168 00000100:00080000:0.0:1787827228.895177:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827228.895182:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827228.895189:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827228.895204:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564121856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827228.895216:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827228.895224:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787827228.895226:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787827228.895229:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827228.895235:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2168 00000100:00080000:0.0:1787827228.895241:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827228.895246:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827228.895251:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827228.895267:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564121984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827228.895275:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827228.895277:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827228.895279:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827228.895284:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2168 00000100:00080000:0.0:1787827228.895288:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827228.895289:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827228/real 1787827228] req@ffff976882aef0c0 x1874670564121600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827228.895295:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827228.895309:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827228.895322:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564122112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827228.895334:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564121600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827228.895350:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827228.895356:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827228.895363:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564121600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827228.895392:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564121728:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827228.895419:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827228.895434:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827228/real 1787827228] req@ffff976882aee940 x1874670564121728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827228.895459:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564121728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827228.895469:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827228.895472:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827228.895476:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564121728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827228.895483:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564121856:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827228.895507:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827228.895519:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827228/real 1787827228] req@ffff976882aee580 x1874670564121856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827228.895527:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564121856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827228.895533:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827228.895536:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827228.895538:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564121856:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827228.895542:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564121984:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827228.895558:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827228.895565:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827228/real 1787827228] req@ffff976882aeed00 x1874670564121984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827228.895572:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564121984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827228.895578:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827228.895580:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827228.895582:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564121984:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827228.895586:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564122112:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827228.895623:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827228.895632:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827228/real 1787827228] req@ffff976882aec3c0 x1874670564122112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827228.895640:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564122112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827228.895645:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827228.895648:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827228.895650:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564122112:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827229.150907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827229.150907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827229.150968:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827229.918963:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827230.174893:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827230.174926:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827230.174955:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827230.942938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827231.198878:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827231.198957:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827231.199020:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827231.966932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787827232.222986:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827232.222987:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827232.223007:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827232.990935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827233.246930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827233.246930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827233.246993:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827234.014948:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827234.014953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827234.014967:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827234.014981:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827234.014994:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2173 00000100:00080000:1.0:1787827234.015011:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827234.015020:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827234.015071:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564122240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827234.015108:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827234.015112:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827234.015114:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827234.015119:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2173 00000100:00080000:1.0:1787827234.015125:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827234.015130:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827234.015144:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827234.015145:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564122240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827234.015180:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564122368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827234.015199:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827234.015203:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827234.015205:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827234.015210:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2173 00000100:00080000:1.0:1787827234.015216:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827234.015223:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787827234.015228:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827234.015237:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827234.015259:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564122496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827234.015267:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827234/real 1787827234] req@ffff976889efcb40 x1874670564122240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827234.015275:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827234.015284:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827234.015288:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827234.015312:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2173 00000100:00080000:1.0:1787827234.015318:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827234.015322:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564122240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827234.015328:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827234.015342:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00080000:2.0:1787827234.015348:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827234.015360:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827234.015362:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564122624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827234.015370:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564122240:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827234.015390:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827234.015396:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827234.015399:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827234.015403:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2173 00000100:00080000:1.0:1787827234.015407:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827234.015411:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827234.015416:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564122368:192.168.203.139@tcp:38:kworker.0 00000008:00100000:1.0:1787827234.015419:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827234.015434:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564122752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827234.015460:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827234.015479:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827234/real 1787827234] req@ffff976889efe1c0 x1874670564122368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827234.015493:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564122368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827234.015503:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827234.015507:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827234.015511:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564122368:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827234.015518:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564122496:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827234.015543:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827234.015556:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827234/real 1787827234] req@ffff976889efe580 x1874670564122496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827234.015585:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564122496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827234.015594:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827234.015598:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827234.015601:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564122496:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827234.015608:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564122624:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827234.015631:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827234.015643:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827234/real 1787827234] req@ffff976889eff480 x1874670564122624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827234.015656:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564122624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827234.015678:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827234.015682:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827234.015686:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564122624:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827234.015706:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564122752:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827234.015727:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827234.015740:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827234/real 1787827234] req@ffff976889efc780 x1874670564122752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827234.015753:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564122752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827234.015762:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827234.015765:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827234.015769:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564122752:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827234.270933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827234.270948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827234.270972:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827235.038888:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827235.294921:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827235.294924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827235.295124:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827236.062939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827236.318924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827236.318925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827236.318957:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827237.086906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827237.342930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827237.342930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827237.342988:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827238.110921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827238.366922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827238.366922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827238.366959:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827239.134928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827239.134939:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827239.134946:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827239.134949:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827239.134957:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2179 00000100:00080000:0.0:1787827239.134966:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827239.134972:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827239.135000:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564122880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827239.135020:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827239.135023:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827239.135024:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827239.135027:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2179 00000100:00080000:0.0:1787827239.135031:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827239.135036:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827239.135046:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827239.135052:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564122880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827239.135064:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564123008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827239.135078:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827239.135083:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827239.135085:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827239.135089:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2179 00000100:00080000:0.0:1787827239.135092:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827239.135096:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827239.135100:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827239.135106:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827239.135117:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564123136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827239.135128:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827239.135132:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827239.135134:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827239.135138:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827239/real 1787827239] req@ffff976882aed680 x1874670564122880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827239.135143:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2179 00000100:00100000:2.0:1787827239.135153:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564122880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827239.135164:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827239.135166:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827239.135171:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:0.0:1787827239.135174:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827239.135182:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827239.135184:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564122880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827239.135204:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564123264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827239.135210:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564123008:192.168.203.139@tcp:38:kworker.0 00000100:00080000:0.0:1787827239.135225:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827239.135228:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827239.135230:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787827239.135234:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787827239.135237:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2179 00000100:00080000:0.0:1787827239.135243:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827239.135247:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827239.135250:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827239/real 1787827239] req@ffff976882aec3c0 x1874670564123008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827239.135255:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827239.135268:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564123008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827239.135279:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787827239.135285:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564123392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827239.135286:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827239.135295:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564123008:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827239.135310:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564123136:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827239.135333:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827239.135345:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827239/real 1787827239] req@ffff976882aeed00 x1874670564123136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827239.135359:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564123136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827239.135368:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827239.135371:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827239.135375:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564123136:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827239.135390:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564123264:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827239.135406:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827239.135414:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827239/real 1787827239] req@ffff976882aee580 x1874670564123264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827239.135421:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564123264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827239.135427:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827239.135429:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827239.135431:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564123264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827239.135435:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564123392:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827239.135445:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827239.135450:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827239/real 1787827239] req@ffff976882aee940 x1874670564123392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827239.135457:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564123392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827239.135462:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827239.135464:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827239.135466:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564123392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827239.390933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827239.390955:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827239.391038:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827240.158949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827240.414922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827240.414928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827240.414958:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827241.182935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827241.438967:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827241.438977:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827241.439041:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827242.206951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827242.462900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827242.462905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827242.462935:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827243.230924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827243.486933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827243.486947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827243.486997:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827244.254951:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827244.254956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827244.254969:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827244.254975:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827244.254986:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2184 00000100:00080000:3.0:1787827244.254998:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827244.255005:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827244.255048:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564123520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827244.255074:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827244.255077:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827244.255079:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827244.255082:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2184 00000100:00080000:3.0:1787827244.255085:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827244.255088:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827244.255099:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827244.255114:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564123520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827244.255118:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564123648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827244.255135:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827244.255140:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827244.255142:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827244.255148:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2184 00000100:00080000:3.0:1787827244.255152:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827244.255156:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827244.255161:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827244.255172:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564123776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827244.255180:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827244.255182:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827244.255184:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827244.255187:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2184 00000100:00080000:3.0:1787827244.255189:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827244.255192:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827244.255196:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827244.255207:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564123904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827244.255216:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827244.255220:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827244.255222:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827244.255225:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2184 00000100:00080000:3.0:1787827244.255231:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827244.255237:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564123648:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787827244.255239:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827244.255250:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827244.255271:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564124032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827244.255275:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564123776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827244.255314:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564123904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827244.255335:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564124032:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827244.510934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827244.510949:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827244.510992:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827245.278981:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827245.534967:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827245.535125:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827245.535146:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827246.302924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827246.558890:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787827246.558890:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827246.558978:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827247.327006:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:1.0:1787827247.327115:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:1.0:1787827247.327160:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787827247.327179:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827247.327219:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827247.327236:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827247.327252:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827247.327268:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:3.0:1787827247.327315:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827244/real 1787827247] req@ffff97688cb5cf00 x1874670564124032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827247.327343:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564124032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827247.327356:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827247.327362:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827247.327367:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564124032:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827247.327376:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827244/real 1787827247] req@ffff97688cb5e1c0 x1874670564123648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827247.327392:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564123648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827247.327402:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827247.327407:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827247.327410:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564123648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827247.327421:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827244/real 1787827247] req@ffff97688cb5c3c0 x1874670564123776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827247.327435:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564123776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827247.327445:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827247.327450:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827247.327453:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564123776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827247.327461:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827244/real 1787827247] req@ffff97688cb5f840 x1874670564123904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827247.327489:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564123904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827247.327499:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827247.327503:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827247.327507:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564123904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827247.327514:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827244/real 1787827247] req@ffff97688cb5c780 x1874670564123520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827247.327529:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564123520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827247.327538:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827247.327542:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827247.327546:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564123520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827247.582943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827247.582976:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827247.583027:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827248.350997:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827248.606910:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827248.606916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827248.606975:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827249.374993:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827249.375002:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827249.375008:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827249.375017:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2189 00000100:00080000:2.0:1787827249.375027:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827249.375034:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827249.375035:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827249.375080:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564124160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827249.375111:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827249.375113:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827249.375115:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827249.375118:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2189 00000100:00080000:2.0:1787827249.375121:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827249.375125:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827249.375134:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827249.375145:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564124288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827249.375157:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827249.375167:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787827249.375168:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564124160:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787827249.375171:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827249.375187:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2189 00000100:00080000:2.0:1787827249.375191:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827249.375196:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827249.375202:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827249.375217:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564124416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827249.375229:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827249.375233:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827249.375235:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827249.375240:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2189 00000100:00080000:2.0:1787827249.375246:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827249.375251:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827249.375257:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787827249.375269:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827249.375274:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564124544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827249.375305:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827249.375310:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827249.375312:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827249.375320:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2189 00000100:00080000:2.0:1787827249.375326:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827249.375330:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827249.375337:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827249.375350:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564124672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827249.375363:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827249/real 1787827249] req@ffff976889cf9e00 x1874670564124160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827249.375392:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564124160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827249.375408:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827249.375414:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827249.375419:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564124160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827249.375455:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564124288:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827249.375504:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827249.375521:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827249/real 1787827249] req@ffff976889cfb480 x1874670564124288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827249.375537:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564124288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827249.375547:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827249.375551:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827249.375555:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564124288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827249.375563:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564124416:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827249.375587:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827249.375599:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827249/real 1787827249] req@ffff976889cfa580 x1874670564124416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827249.375615:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564124416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827249.375626:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827249.375630:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827249.375634:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564124416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827249.375642:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564124544:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827249.375666:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827249.375679:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827249/real 1787827249] req@ffff976889cfa1c0 x1874670564124544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827249.375693:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564124544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827249.375703:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827249.375707:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827249.375711:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564124544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827249.375719:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564124672:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827249.375741:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827249.375753:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827249/real 1787827249] req@ffff976889cf92c0 x1874670564124672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827249.376027:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564124672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827249.376046:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827249.376050:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827249.376054:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564124672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827249.630940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827249.630967:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827249.630993:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827250.399067:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827250.654907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827250.654949:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827250.654987:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827251.422955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827251.678922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827251.678925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827251.678946:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827252.446948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827252.702946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827252.702946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827252.703030:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827253.470955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827253.726940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827253.726940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827253.727006:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827254.494916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827254.494935:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827254.494943:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827254.494948:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827254.494959:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2194 00000100:00080000:1.0:1787827254.494971:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827254.494979:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827254.495021:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564124800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827254.495053:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827254.495057:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827254.495059:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827254.495063:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2194 00000100:00080000:1.0:1787827254.495066:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827254.495069:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827254.495080:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827254.495095:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564124800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827254.495097:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564124928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827254.495119:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827254.495124:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827254.495127:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827254.495135:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2194 00000100:00080000:1.0:1787827254.495144:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827254.495151:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827254.495158:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827254.495165:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827254.495172:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564125056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827254.495185:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827254.495188:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827254.495190:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827254.495195:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2194 00000100:00080000:1.0:1787827254.495201:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827254.495207:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827254.495214:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827254.495221:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827254/real 1787827254] req@ffff976889efc780 x1874670564124800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827254.495229:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564125184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827254.495242:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827254.495248:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564124800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827254.495249:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827254.495258:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827254.495272:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2194 00000100:00080000:2.0:1787827254.495276:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827254.495277:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827254.495282:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:1.0:1787827254.495283:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827254.495295:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827254.495295:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564124800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827254.495321:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564125312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827254.495337:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564124928:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827254.495361:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827254.495372:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827254/real 1787827254] req@ffff976889eff480 x1874670564124928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827254.495382:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564124928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827254.495388:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827254.495391:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827254.495393:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564124928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827254.495397:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564125056:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827254.495409:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827254.495414:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827254/real 1787827254] req@ffff976889efe580 x1874670564125056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827254.495443:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564125056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827254.495449:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827254.495451:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827254.495454:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564125056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827254.495460:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564125184:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827254.495479:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827254.495492:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827254/real 1787827254] req@ffff976889efe1c0 x1874670564125184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827254.495502:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564125184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827254.495509:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827254.495512:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827254.495517:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564125184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827254.495545:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564125312:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827254.495582:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827254.495593:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827254/real 1787827254] req@ffff976889efcb40 x1874670564125312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827254.495601:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564125312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827254.495607:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827254.495610:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827254.495612:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564125312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827254.750925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827254.750925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827254.750999:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827255.518920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827255.774962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827255.774990:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827255.775027:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827256.542927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827256.798932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827256.798938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827256.799077:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827257.566908:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827257.822987:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827257.822988:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827257.823066:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827258.590922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827258.846962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827258.846977:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827258.847037:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827259.614917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827259.614928:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827259.614936:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827259.614940:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827259.614948:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2199 00000100:00080000:1.0:1787827259.614958:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827259.614972:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827259.615007:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564125440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827259.615029:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827259.615031:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827259.615033:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827259.615036:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2199 00000100:00080000:1.0:1787827259.615040:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827259.615043:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827259.615054:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827259.615069:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564125568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827259.615080:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827259.615085:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827259.615088:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827259.615093:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2199 00000100:00080000:1.0:1787827259.615097:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827259.615101:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827259.615107:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827259.615120:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564125440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827259.615121:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564125696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827259.615145:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827259.615149:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827259.615151:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827259.615157:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2199 00000100:00080000:1.0:1787827259.615162:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827259.615166:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827259.615171:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827259.615183:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564125824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827259.615190:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827259.615192:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827259.615193:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827259.615197:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2199 00000100:00080000:1.0:1787827259.615201:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827259.615203:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827259.615206:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827259.615218:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564125952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827259.615233:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827259.615290:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827259/real 1787827259] req@ffff976889efc3c0 x1874670564125440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827259.615312:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564125440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827259.615327:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827259.615332:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827259.615337:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564125440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827259.615346:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564125568:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827259.615376:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827259.615409:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827259/real 1787827259] req@ffff976889efed00 x1874670564125568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827259.615423:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564125568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827259.615433:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827259.615437:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827259.615441:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564125568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827259.615473:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564125696:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827259.615496:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827259.615508:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827259/real 1787827259] req@ffff976889effc00 x1874670564125696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827259.615521:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564125696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827259.615531:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827259.615535:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827259.615539:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564125696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827259.615547:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564125824:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827259.615567:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827259.615579:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827259/real 1787827259] req@ffff976889eff0c0 x1874670564125824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827259.615592:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564125824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827259.615600:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827259.615604:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827259.615608:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564125824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827259.615614:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564125952:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827259.615635:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827259.615646:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827259/real 1787827259] req@ffff976889efe940 x1874670564125952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827259.615658:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564125952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827259.615666:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827259.615669:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827259.615673:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564125952:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827259.870934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827259.870950:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827259.871027:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827260.638999:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827260.894973:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827260.894980:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827260.895038:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827261.662923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827261.918937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827261.918944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827261.919002:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827262.687016:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827262.942978:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827262.942979:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827262.943045:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827263.710931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827263.966897:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827263.966944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827263.967019:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827264.734973:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827264.734982:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827264.734986:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827264.734995:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2204 00000100:00100000:1.0:1787827264.735004:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827264.735005:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827264.735016:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827264.735050:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564126080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827264.735075:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827264.735077:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827264.735079:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827264.735081:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2204 00000100:00080000:0.0:1787827264.735084:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827264.735086:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827264.735094:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827264.735105:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564126208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827264.735112:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827264.735115:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827264.735117:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827264.735121:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2204 00000100:00080000:0.0:1787827264.735124:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827264.735132:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564126080:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787827264.735133:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827264.735145:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827264.735154:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564126336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827264.735162:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827264.735165:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827264.735166:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827264.735170:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2204 00000100:00080000:0.0:1787827264.735172:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827264.735183:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827264.735187:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827264.735196:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564126464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827264.735202:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827264.735205:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827264.735207:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827264.735211:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2204 00000100:00080000:0.0:1787827264.735213:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827264.735217:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827264.735221:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827264.735228:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827264.735239:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564126592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827264.735280:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827264/real 1787827264] req@ffff976882aef0c0 x1874670564126080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827264.735299:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564126080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827264.735309:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827264.735313:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827264.735316:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564126080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827264.735322:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564126208:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827264.735340:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827264.735349:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827264/real 1787827264] req@ffff976882aee940 x1874670564126208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827264.735371:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564126208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827264.735378:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827264.735380:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827264.735382:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564126208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827264.735407:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564126336:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827264.735421:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827264.735427:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827264/real 1787827264] req@ffff976882aee580 x1874670564126336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827264.735436:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564126336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827264.735442:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827264.735444:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827264.735447:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564126336:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827264.735450:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564126464:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827264.735461:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827264.735466:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827264/real 1787827264] req@ffff976882aeed00 x1874670564126464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827264.735474:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564126464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827264.735479:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827264.735481:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827264.735483:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564126464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827264.735487:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564126592:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827264.735496:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827264.735502:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827264/real 1787827264] req@ffff976882aec3c0 x1874670564126592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827264.735508:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564126592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827264.735518:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827264.735520:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827264.735523:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564126592:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827264.990942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827264.990974:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827264.991016:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827265.758893:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827266.014968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827266.014973:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827266.014991:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827266.782929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827267.038900:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827267.038913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827267.038995:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827267.806947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827268.062910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827268.062936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827268.062947:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827268.830947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827269.086930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827269.086931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827269.086978:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827269.854923:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827269.854923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827269.854933:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827269.854938:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827269.854947:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2209 00000100:00080000:0.0:1787827269.854956:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827269.854962:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827269.854998:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564126720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827269.855026:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827269.855029:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827269.855032:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827269.855036:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2209 00000100:00080000:0.0:1787827269.855040:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827269.855045:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827269.855055:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827269.855068:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564126720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827269.855074:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564126848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827269.855087:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827269.855091:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827269.855095:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827269.855099:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2209 00000100:00080000:0.0:1787827269.855104:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827269.855108:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827269.855113:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827269.855126:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564126976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827269.855127:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787827269.855137:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827269.855140:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827269.855142:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827269.855146:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2209 00000100:00080000:0.0:1787827269.855148:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827269.855151:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827269.855155:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827269.855165:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564127104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827269.855170:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827269/real 1787827269] req@ffff976882aed680 x1874670564126720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827269.855174:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827269.855179:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827269.855181:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827269.855188:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2209 00000100:00080000:0.0:1787827269.855192:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827269.855195:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827269.855198:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827269.855205:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564126720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827269.855207:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564127232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827269.855241:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827269.855247:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827269.855253:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564126720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827269.855278:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564126848:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827269.855300:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827269.855310:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827269/real 1787827269] req@ffff976882aec3c0 x1874670564126848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827269.855336:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564126848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827269.855343:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827269.855345:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827269.855348:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564126848:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827269.855352:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564126976:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827269.855370:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827269.855378:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827269/real 1787827269] req@ffff976882aeed00 x1874670564126976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827269.855386:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564126976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827269.855391:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827269.855393:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827269.855396:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564126976:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827269.855399:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564127104:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827269.855410:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827269.855415:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827269/real 1787827269] req@ffff976882aee580 x1874670564127104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827269.855422:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564127104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827269.855427:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827269.855428:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827269.855431:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564127104:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827269.855435:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564127232:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827269.855444:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827269.855449:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827269/real 1787827269] req@ffff976882aee940 x1874670564127232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827269.855457:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564127232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827269.855465:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827269.855468:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827269.855471:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564127232:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827270.110881:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827270.110890:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827270.110958:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827270.878925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827271.134918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827271.134939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827271.134953:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827271.903001:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827272.158914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827272.158935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827272.159007:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827272.926946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827273.182942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827273.182943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827273.183002:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827273.950925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827274.206940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827274.206955:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827274.207035:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827274.974905:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827274.974906:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827274.974915:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827274.974919:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827274.974928:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2214 00000100:00080000:0.0:1787827274.974938:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827274.974944:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827274.974975:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564127360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827274.974997:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827274.974999:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827274.975001:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827274.975003:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2214 00000100:00080000:0.0:1787827274.975005:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827274.975008:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827274.975015:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827274.975025:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564127488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827274.975033:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827274.975037:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564127360:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787827274.975038:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827274.975042:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827274.975048:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2214 00000100:00080000:0.0:1787827274.975051:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827274.975055:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827274.975060:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827274.975073:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564127616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827274.975083:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827274.975087:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827274.975090:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827274.975094:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2214 00000100:00080000:0.0:1787827274.975098:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827274.975103:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787827274.975107:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827274.975113:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827274.975124:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564127744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827274.975131:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827274.975134:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827274.975136:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827274.975141:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2214 00000100:00100000:2.0:1787827274.975145:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827274/real 1787827274] req@ffff976882aef0c0 x1874670564127360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827274.975147:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827274.975157:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827274.975164:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827274.975165:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564127360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827274.975177:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827274.975182:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827274.975185:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564127872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827274.975188:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564127360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827274.975212:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564127488:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827274.975231:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827274.975240:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827274/real 1787827274] req@ffff976882aee940 x1874670564127488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827274.975248:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564127488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827274.975254:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827274.975256:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827274.975258:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564127488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827274.975263:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564127616:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827274.975273:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827274.975295:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827274/real 1787827274] req@ffff976882aee580 x1874670564127616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827274.975305:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564127616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827274.975311:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827274.975313:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827274.975315:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564127616:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827274.975319:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564127744:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827274.975333:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827274.975341:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827274/real 1787827274] req@ffff976882aeed00 x1874670564127744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827274.975348:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564127744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827274.975353:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827274.975355:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827274.975357:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564127744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827274.975360:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564127872:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827274.975371:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827274.975376:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827274/real 1787827274] req@ffff976882aec3c0 x1874670564127872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827274.975383:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564127872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827274.975388:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827274.975390:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827274.975392:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564127872:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827275.230949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827275.230963:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827275.231019:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827275.998903:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827276.254931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827276.254947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827276.254979:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827277.022900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827277.278891:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827277.278900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827277.278990:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827278.046918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827278.302923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827278.302927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827278.302975:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827279.070881:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827279.326931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827279.326951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827279.327008:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827280.094971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827280.095000:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827280.095007:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827280.095012:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827280.095022:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2219 00000100:00080000:3.0:1787827280.095034:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827280.095041:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827280.095091:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564128000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827280.095125:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827280.095129:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827280.095132:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827280.095137:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2219 00000100:00080000:3.0:1787827280.095141:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827280.095148:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827280.095161:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827280.095175:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564128128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827280.095183:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827280.095186:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827280.095189:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827280.095192:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2219 00000100:00080000:3.0:1787827280.095195:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827280.095202:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827280.095205:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564128000:192.168.203.139@tcp:250:kworker.0 00000008:00100000:3.0:1787827280.095207:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827280.095221:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564128256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827280.095230:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827280.095233:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827280.095234:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827280.095237:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2219 00000100:00080000:3.0:1787827280.095240:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827280.095242:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827280.095263:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827280.095279:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564128384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827280.095290:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827280.095294:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827280.095296:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827280.095301:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2219 00000100:00080000:3.0:1787827280.095306:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827280.095312:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827280.095318:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827280.095326:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827280.095347:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564128512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827280.095360:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827280/real 1787827280] req@ffff97688cb5c780 x1874670564128000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827280.095381:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564128000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827280.095394:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827280.095399:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827280.095404:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564128000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827280.095413:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564128128:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827280.095438:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827280.095448:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827280/real 1787827280] req@ffff97688cb5f840 x1874670564128128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827280.095457:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564128128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827280.095463:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827280.095465:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827280.095467:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564128128:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827280.095491:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564128256:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827280.095504:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827280.095510:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827280/real 1787827280] req@ffff97688cb5c3c0 x1874670564128256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827280.095522:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564128256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827280.095528:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827280.095530:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827280.095532:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564128256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827280.095536:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564128384:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827280.095547:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827280.095553:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827280/real 1787827280] req@ffff97688cb5e1c0 x1874670564128384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827280.095560:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564128384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827280.095565:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827280.095567:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827280.095569:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564128384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827280.095572:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564128512:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827280.095582:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827280.095588:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827280/real 1787827280] req@ffff97688cb5cf00 x1874670564128512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827280.095603:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564128512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827280.095612:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827280.095615:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827280.095617:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564128512:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827280.350971:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827280.350971:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827280.351044:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827281.118917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827281.374942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827281.374943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827281.375030:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827282.142926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827282.398918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827282.398919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827282.398981:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827283.166895:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827283.422895:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827283.422896:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827283.422974:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827284.190961:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827284.446896:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827284.446899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827284.446949:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827285.214901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827285.214925:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827285.214932:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827285.214937:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827285.214948:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2225 00000100:00080000:3.0:1787827285.214971:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827285.214980:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827285.215018:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564128640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827285.215054:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827285.215057:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827285.215060:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827285.215066:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2225 00000100:00080000:3.0:1787827285.215070:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827285.215078:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827285.215079:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564128640:192.168.203.139@tcp:250:kworker.0 00000008:00100000:3.0:1787827285.215100:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827285.215119:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564128768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827285.215132:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827285.215135:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827285.215138:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827285.215143:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2225 00000100:00080000:3.0:1787827285.215148:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827285.215153:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827285.215157:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827285.215171:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827285.215184:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564128896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827285.215195:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827285.215198:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827285.215200:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827285.215205:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2225 00000100:00100000:2.0:1787827285.215234:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827285/real 1787827285] req@ffff97688cb5d2c0 x1874670564128640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827285.215235:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827285.215246:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827285.215252:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827285.215257:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564128640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827285.215267:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787827285.215269:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564129024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827285.215273:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827285.215280:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564128640:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827285.215291:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827285.215295:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827285.215297:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827285.215301:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2225 00000100:00080000:3.0:1787827285.215305:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827285.215306:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564128768:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787827285.215314:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827285.215320:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827285.215332:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827285.215335:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564129152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827285.215347:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827285/real 1787827285] req@ffff97688cb5cf00 x1874670564128768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827285.215364:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564128768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827285.215384:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827285.215388:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827285.215392:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564128768:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827285.215399:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564128896:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827285.215421:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827285.215430:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827285/real 1787827285] req@ffff97688cb5e1c0 x1874670564128896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827285.215438:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564128896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827285.215444:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827285.215446:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827285.215448:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564128896:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827285.215452:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564129024:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827285.215464:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827285.215470:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827285/real 1787827285] req@ffff97688cb5c3c0 x1874670564129024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827285.215477:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564129024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827285.215482:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827285.215484:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827285.215487:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564129024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827285.215502:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564129152:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827285.215512:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827285.215518:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827285/real 1787827285] req@ffff97688cb5f840 x1874670564129152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827285.215525:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564129152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827285.215531:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827285.215533:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827285.215535:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564129152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827285.470896:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827285.470908:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827285.470970:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827286.238896:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827286.494929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827286.494930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827286.494962:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827287.262927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827287.518889:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827287.518905:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827287.518976:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827288.286905:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827288.542895:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827288.542901:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827288.542967:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827289.310938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827289.566905:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827289.566915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827289.566947:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827290.334920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827290.334944:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827290.334948:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827290.334951:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827290.334958:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2230 00000100:00080000:1.0:1787827290.334967:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827290.334975:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827290.335008:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564129280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827290.335038:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827290.335041:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827290.335043:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827290.335047:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2230 00000100:00080000:1.0:1787827290.335049:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827290.335052:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827290.335060:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827290.335072:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564129408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827290.335078:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564129280:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827290.335082:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827290.335088:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827290.335089:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827290.335094:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2230 00000100:00080000:1.0:1787827290.335097:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827290.335099:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827290.335103:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827290.335114:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564129536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827290.335120:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827290.335123:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827290.335124:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827290.335128:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2230 00000100:00080000:1.0:1787827290.335131:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827290.335134:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827290.335137:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827290.335144:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564129664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827290.335150:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827290.335157:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1787827290.335157:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787827290.335160:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827290.335165:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2230 00000100:00080000:1.0:1787827290.335169:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827290.335182:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827290.335187:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827290.335197:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564129792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827290.335277:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827290/real 1787827290] req@ffff976889efe940 x1874670564129280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827290.335299:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564129280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827290.335437:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827290.335443:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827290.335448:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564129280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827290.335527:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564129408:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827290.335562:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827290.335586:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827290/real 1787827290] req@ffff976889eff0c0 x1874670564129408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827290.335602:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564129408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827290.335612:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827290.335616:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827290.335627:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564129408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827290.335635:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564129536:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827290.335669:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827290.335683:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827290/real 1787827290] req@ffff976889effc00 x1874670564129536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827290.335703:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564129536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827290.335712:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827290.335716:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827290.335720:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564129536:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827290.335726:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564129664:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827290.335749:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827290.335762:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827290/real 1787827290] req@ffff976889efed00 x1874670564129664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827290.335775:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564129664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827290.335785:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827290.335788:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827290.335792:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564129664:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827290.335798:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564129792:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827290.335868:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827290.335886:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827290/real 1787827290] req@ffff976889efc3c0 x1874670564129792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827290.335901:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564129792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827290.335921:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827290.335926:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827290.335929:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564129792:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827290.590932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827290.590932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827290.591039:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827291.358868:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827291.614927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827291.614928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827291.614965:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827292.382898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827292.638899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827292.638902:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827292.638966:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827293.406899:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827293.662891:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827293.662912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827293.662921:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827294.430875:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827294.686912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827294.686961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827294.686993:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827295.454885:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827295.454895:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827295.454900:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827295.454910:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2235 00000100:00080000:1.0:1787827295.454920:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827295.454938:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827295.454974:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564129920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827295.455007:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827295.455007:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827295.455014:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827295.455016:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827295.455020:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2235 00000100:00080000:1.0:1787827295.455024:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827295.455028:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827295.455036:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827295.455042:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564129920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827295.455048:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564130048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827295.455056:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827295.455059:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827295.455060:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827295.455062:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2235 00000100:00080000:1.0:1787827295.455064:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827295.455074:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827295.455077:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827295.455086:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564130176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827295.455093:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827295.455096:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827295.455098:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827295.455101:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2235 00000100:00080000:1.0:1787827295.455103:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827295.455105:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:3.0:1787827295.455109:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787827295.455111:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827295.455121:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564130304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827295.455127:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827295.455152:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827295.455154:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827295.455157:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2235 00000100:00080000:1.0:1787827295.455160:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827295.455164:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827295.455168:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827295.455169:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827295/real 1787827295] req@ffff976889efcb40 x1874670564129920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827295.455181:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564130432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827295.455188:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564129920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827295.455214:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827295.455218:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827295.455221:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564129920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827295.455242:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564130048:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827295.455260:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827295.455269:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827295/real 1787827295] req@ffff976889efe1c0 x1874670564130048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827295.455277:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564130048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827295.455283:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827295.455286:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827295.455289:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564130048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827295.455292:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564130176:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827295.455303:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827295.455309:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827295/real 1787827295] req@ffff976889efe580 x1874670564130176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827295.455316:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564130176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827295.455321:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827295.455323:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827295.455325:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564130176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827295.455328:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564130304:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827295.455338:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827295.455344:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827295/real 1787827295] req@ffff976889eff480 x1874670564130304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827295.455351:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564130304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827295.455356:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827295.455358:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827295.455360:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564130304:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827295.455365:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564130432:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827295.455383:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827295.455394:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827295/real 1787827295] req@ffff976889efc780 x1874670564130432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827295.455407:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564130432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827295.455415:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827295.455418:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827295.455422:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564130432:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827295.710897:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827295.710897:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827295.710905:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827296.478921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827296.734901:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827296.734941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827296.734942:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827297.503021:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827297.758914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827297.758972:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827297.759003:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827298.526889:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827298.782886:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827298.782899:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827298.782916:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827299.550935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827299.806902:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827299.806931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827299.806932:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827300.574921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827300.574942:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827300.574953:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827300.574958:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827300.574968:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2240 00000100:00080000:1.0:1787827300.574978:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827300.574987:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827300.575023:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564130560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827300.575056:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827300.575060:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827300.575062:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827300.575066:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2240 00000100:00080000:1.0:1787827300.575070:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827300.575074:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827300.575084:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827300.575099:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564130688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827300.575128:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827300.575132:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827300.575134:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827300.575138:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2240 00000100:00080000:1.0:1787827300.575142:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827300.575149:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564130560:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827300.575150:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827300.575161:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827300.575178:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564130816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827300.575190:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827300.575194:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827300.575196:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827300.575201:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2240 00000100:00080000:1.0:1787827300.575213:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827300.575220:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:3.0:1787827300.575221:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787827300.575230:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827300.575243:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564130944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827300.575254:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827300.575258:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827300.575260:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827300.575264:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2240 00000100:00100000:3.0:1787827300.575269:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827300/real 1787827300] req@ffff976889efc780 x1874670564130560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827300.575270:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827300.575280:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827300.575286:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827300.575291:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564130560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827300.575301:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787827300.575304:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564131072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827300.575306:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827300.575313:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564130560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827300.575325:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564130688:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827300.575347:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827300.575358:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827300/real 1787827300] req@ffff976889eff480 x1874670564130688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827300.575365:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564130688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827300.575371:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827300.575373:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827300.575375:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564130688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827300.575394:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564130816:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827300.575405:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827300.575411:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827300/real 1787827300] req@ffff976889efe580 x1874670564130816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827300.575419:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564130816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827300.575425:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827300.575427:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827300.575429:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564130816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827300.575432:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564130944:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827300.575442:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827300.575447:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827300/real 1787827300] req@ffff976889efe1c0 x1874670564130944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827300.575459:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564130944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827300.575464:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827300.575465:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827300.575467:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564130944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827300.575470:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564131072:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827300.575479:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827300.575485:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827300/real 1787827300] req@ffff976889efcb40 x1874670564131072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827300.575491:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564131072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827300.575506:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827300.575508:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827300.575510:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564131072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827300.830922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827300.830922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827300.830946:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827301.599041:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827301.854944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827301.854964:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827301.854979:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827302.622912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827302.878918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827302.878918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827302.878958:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827303.646921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827303.902930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827303.902942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827303.902962:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827304.670923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827304.926899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827304.926905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827304.926940:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827305.694915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827305.694921:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827305.694927:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827305.694930:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827305.694939:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2245 00000100:00080000:1.0:1787827305.694959:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827305.694966:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827305.695000:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564131200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827305.695025:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827305.695028:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827305.695029:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827305.695032:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2245 00000100:00080000:1.0:1787827305.695036:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827305.695039:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827305.695047:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827305.695054:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564131200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827305.695060:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564131328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827305.695079:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827305.695083:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827305.695084:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827305.695087:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2245 00000100:00080000:1.0:1787827305.695089:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827305.695092:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827305.695097:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827305.695104:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564131456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827305.695123:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827305.695125:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827305.695127:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827305.695130:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2245 00000100:00080000:1.0:1787827305.695132:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827305.695134:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827305.695137:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827305.695144:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564131584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827305.695150:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827305.695154:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1787827305.695156:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787827305.695157:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827305.695162:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2245 00000100:00080000:1.0:1787827305.695165:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827305.695168:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827305.695172:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827305.695179:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564131712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827305.695201:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827305/real 1787827305] req@ffff976889efc3c0 x1874670564131200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827305.695217:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564131200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827305.695224:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827305.695228:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827305.695231:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564131200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827305.695250:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564131328:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827305.695271:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827305.695284:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827305/real 1787827305] req@ffff976889efed00 x1874670564131328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827305.695297:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564131328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827305.695306:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827305.695309:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827305.695313:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564131328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827305.695320:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564131456:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827305.695352:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827305.695365:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827305/real 1787827305] req@ffff976889effc00 x1874670564131456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827305.695379:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564131456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827305.695395:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827305.695399:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827305.695402:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564131456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827305.695409:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564131584:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827305.695430:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827305.695441:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827305/real 1787827305] req@ffff976889eff0c0 x1874670564131584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827305.695454:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564131584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827305.695463:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827305.695467:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827305.695470:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564131584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827305.695476:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564131712:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827305.695491:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827305.695498:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827305/real 1787827305] req@ffff976889efe940 x1874670564131712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827305.695506:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564131712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827305.695511:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827305.695513:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827305.695516:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564131712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827305.950923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827305.950923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827305.950941:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827306.719021:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827306.974935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827306.974968:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827306.974968:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827307.742971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827307.998928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827307.998963:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827307.998971:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827308.766953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787827309.022958:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827309.022967:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827309.022969:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827309.790956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827310.046945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827310.046945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827310.046983:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827310.814973:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827310.814979:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827310.814990:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827310.814996:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827310.815009:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2250 00000100:00080000:1.0:1787827310.815026:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827310.815050:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827310.815106:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564131840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827310.815142:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827310.815146:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827310.815148:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827310.815154:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2250 00000100:00080000:1.0:1787827310.815158:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827310.815164:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827310.815176:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827310.815187:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564131840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827310.815195:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564131968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827310.815209:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827310.815213:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827310.815215:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827310.815220:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2250 00000100:00080000:1.0:1787827310.815224:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827310.815229:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827310.815235:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827310.815250:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564132096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827310.815261:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827310.815265:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827310.815267:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827310.815272:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2250 00000100:00080000:1.0:1787827310.815276:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827310.815281:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827310.815287:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827310.815300:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564132224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827310.815306:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564131968:192.168.203.139@tcp:38:kworker.0 00000100:00080000:1.0:1787827310.815320:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827310.815325:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827310.815328:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827310.815334:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2250 00000100:00080000:1.0:1787827310.815339:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827310.815341:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564132096:192.168.203.139@tcp:38:kworker.0 00000100:00080000:1.0:1787827310.815348:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827310.815355:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827310.815367:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564132224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827310.815373:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564132352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827310.815391:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564132352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827311.070904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827311.070904:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827311.070933:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827311.839069:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827312.094971:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827312.094971:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827312.095034:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827312.862977:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827313.118943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827313.118944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827313.119094:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827313.887084:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:1.0:1787827313.887102:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:1.0:1787827313.887144:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787827313.887162:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827313.887197:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827313.887209:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827313.887219:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827313.887227:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:3.0:1787827313.887240:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827310/real 1787827313] req@ffff976889efed00 x1874670564132224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827313.887268:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564132224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827313.887281:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827313.887286:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827313.887291:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564132224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827313.887299:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827310/real 1787827313] req@ffff976889eff0c0 x1874670564131968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827313.887313:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564131968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827313.887321:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827313.887330:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827313.887345:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564131968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827313.887353:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827310/real 1787827313] req@ffff976889effc00 x1874670564132096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827313.887365:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564132096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827313.887374:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827313.887378:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827313.887381:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564132096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827313.887387:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827310/real 1787827313] req@ffff976889efe940 x1874670564131840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827313.887400:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564131840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827313.887408:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827313.887411:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827313.887414:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564131840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827313.887447:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827310/real 1787827313] req@ffff976889efc3c0 x1874670564132352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827313.887462:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564132352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827313.887470:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827313.887474:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827313.887477:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564132352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827314.142930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827314.142945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827314.142973:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827314.910985:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827315.166940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827315.166940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827315.166963:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827315.934907:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827315.934915:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827315.934919:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827315.934927:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2255 00000100:00080000:3.0:1787827315.934936:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827315.934942:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827315.934951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787827315.934973:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564132480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827315.935008:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827315.935012:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827315.935015:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827315.935020:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2255 00000100:00080000:3.0:1787827315.935024:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827315.935031:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564132480:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827315.935032:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827315.935047:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827315.935067:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564132608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827315.935079:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827315.935083:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827315.935085:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827315.935090:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2255 00000100:00080000:3.0:1787827315.935094:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827315.935101:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:2.0:1787827315.935102:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:3.0:1787827315.935108:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827315.935121:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564132736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827315.935129:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827315.935131:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827315.935132:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827315.935136:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2255 00000100:00080000:3.0:1787827315.935139:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827315.935142:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827315.935146:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827315.935150:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827315/real 1787827315] req@ffff97688cb5c780 x1874670564132480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827315.935156:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564132864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827315.935178:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827315.935180:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827315.935182:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827315.935187:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564132480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827315.935192:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2255 00000100:00080000:3.0:1787827315.935198:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827315.935207:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:2.0:1787827315.935211:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787827315.935214:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827315.935222:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827315.935229:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564132480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827315.935231:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564132992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827315.935264:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564132608:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827315.935294:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827315.935308:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827315/real 1787827315] req@ffff97688cb5cb40 x1874670564132608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827315.935321:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564132608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827315.935330:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827315.935333:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827315.935337:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564132608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827315.935343:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564132736:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827315.935364:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827315.935376:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827315/real 1787827315] req@ffff97688cb5c000 x1874670564132736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827315.935388:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564132736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827315.935396:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827315.935399:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827315.935403:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564132736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827315.935408:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564132864:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827315.935430:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827315.935441:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827315/real 1787827315] req@ffff97688cb5d680 x1874670564132864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827315.935452:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564132864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827315.935459:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827315.935463:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827315.935465:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564132864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827315.935470:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564132992:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827315.935487:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827315.935497:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827315/real 1787827315] req@ffff97688cb5f0c0 x1874670564132992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827315.935508:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564132992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827315.935516:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827315.935519:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827315.935522:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564132992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827316.190943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827316.190943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827316.190952:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827316.959231:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827317.214922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827317.214922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827317.214956:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827317.982925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827318.238923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827318.238923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827318.238959:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827319.006940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827319.262992:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827319.262992:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827319.263044:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827320.031011:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827320.286929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827320.286931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827320.286956:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827321.054950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827321.054991:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827321.055018:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827321.055023:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827321.055036:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2260 00000100:00080000:3.0:1787827321.055051:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827321.055062:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827321.055117:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564133120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827321.055183:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827321.055186:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827321.055189:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827321.055193:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2260 00000100:00080000:3.0:1787827321.055197:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827321.055202:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827321.055215:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827321.055247:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564133248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827321.055260:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564133120:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827321.055265:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827321.055269:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827321.055271:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827321.055275:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2260 00000100:00080000:3.0:1787827321.055277:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827321.055280:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827321.055284:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827321.055293:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564133376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827321.055317:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827321.055320:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827321.055322:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827321.055326:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2260 00000100:00080000:3.0:1787827321.055342:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827321.055346:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827321.055351:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827321.055366:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564133504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827321.055372:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827321.055401:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827321.055405:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827321.055408:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827321.055415:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2260 00000100:00080000:3.0:1787827321.055422:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827321.055427:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827321.055434:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827321.055468:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827321/real 1787827321] req@ffff97688cb5de00 x1874670564133120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827321.055470:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564133632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827321.055491:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564133120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827321.055508:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827321.055513:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827321.055520:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564133120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827321.055554:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564133248:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827321.055576:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827321.055587:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827321/real 1787827321] req@ffff97688cb5f0c0 x1874670564133248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827321.055602:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564133248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827321.055609:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827321.055611:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827321.055614:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564133248:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827321.055618:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564133376:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827321.055631:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827321.055638:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827321/real 1787827321] req@ffff97688cb5d680 x1874670564133376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827321.055646:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564133376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827321.055651:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827321.055653:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827321.055655:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564133376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827321.055659:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564133504:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827321.055671:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827321.055677:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827321/real 1787827321] req@ffff97688cb5c000 x1874670564133504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827321.055685:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564133504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827321.055695:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827321.055697:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827321.055699:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564133504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827321.055702:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564133632:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827321.055712:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827321.055718:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827321/real 1787827321] req@ffff97688cb5cb40 x1874670564133632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827321.055724:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564133632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827321.055730:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827321.055733:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827321.055735:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564133632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827321.310946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827321.310946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827321.311007:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827322.078887:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787827322.334936:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827322.335001:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827322.335005:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827323.102916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827323.358967:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827323.358968:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827323.359036:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827324.126920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827324.382918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827324.382959:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827324.383026:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827325.150900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827325.406982:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827325.406985:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827325.407025:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827326.174960:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827326.174960:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827326.174970:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827326.174976:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827326.174987:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2266 00000100:00080000:3.0:1787827326.174999:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827326.175009:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827326.175064:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564133760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827326.175098:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827326.175102:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827326.175103:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827326.175107:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2266 00000100:00080000:3.0:1787827326.175111:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827326.175115:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827326.175125:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827326.175138:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564133888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827326.175146:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827326.175150:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827326.175152:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827326.175155:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2266 00000100:00080000:3.0:1787827326.175158:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827326.175161:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827326.175165:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827326.175172:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564134016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827326.175184:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564133760:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827326.175186:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827326.175199:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827326.175206:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827326.175210:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2266 00000100:00080000:3.0:1787827326.175212:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827326.175215:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827326.175220:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827326.175231:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564134144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827326.175244:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827326.175247:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827326.175249:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827326.175252:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2266 00000100:00080000:3.0:1787827326.175262:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827326.175266:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827326.175269:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827326.175275:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827326.175286:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564134272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827326.175320:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827326/real 1787827326] req@ffff97688cb5c780 x1874670564133760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827326.175346:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564133760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827326.175356:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827326.175359:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827326.175363:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564133760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827326.175373:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564133888:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827326.175395:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827326.175405:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827326/real 1787827326] req@ffff97688cb5f840 x1874670564133888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827326.175415:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564133888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827326.175427:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827326.175431:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827326.175433:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564133888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827326.175459:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564134016:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827326.175472:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827326.175480:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827326/real 1787827326] req@ffff97688cb5c3c0 x1874670564134016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827326.175487:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564134016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827326.175493:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827326.175495:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827326.175497:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564134016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827326.175502:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564134144:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827326.175525:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827326.175544:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827326/real 1787827326] req@ffff97688cb5e1c0 x1874670564134144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827326.175558:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564134144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827326.175566:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827326.175570:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827326.175572:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564134144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827326.175579:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564134272:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827326.175596:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827326.175612:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827326/real 1787827326] req@ffff97688cb5cf00 x1874670564134272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827326.175626:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564134272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827326.175635:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827326.175638:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827326.175642:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564134272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827326.430969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827326.430985:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827326.431006:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827327.198992:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827327.454911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827327.454959:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827327.455002:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827328.222944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827328.478892:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827328.478919:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827328.478962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827329.246923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827329.502928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827329.502928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827329.502989:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827330.270984:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827330.526996:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827330.526996:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827330.527042:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827331.294912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827331.294944:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827331.294949:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827331.294953:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827331.294960:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2271 00000100:00080000:2.0:1787827331.294969:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827331.294976:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827331.295016:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564134400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827331.295046:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827331.295049:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827331.295050:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827331.295053:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2271 00000100:00080000:2.0:1787827331.295056:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827331.295059:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827331.295067:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827331.295080:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564134528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827331.295085:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564134400:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787827331.295093:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827331.295096:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827331.295098:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827331.295101:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2271 00000100:00080000:2.0:1787827331.295103:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827331.295105:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827331.295110:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827331.295119:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564134656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827331.295126:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827331.295128:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827331.295130:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827331.295132:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2271 00000100:00080000:2.0:1787827331.295135:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827331.295137:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827331.295140:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787827331.295145:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827331.295149:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564134784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827331.295157:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827331.295160:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827331.295162:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827331.295164:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2271 00000100:00080000:2.0:1787827331.295166:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827331.295168:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827331.295171:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827331.295179:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564134912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827331.295183:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827331/real 1787827331] req@ffff976889cfad00 x1874670564134400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827331.295206:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564134400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827331.295217:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827331.295222:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827331.295225:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564134400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827331.295248:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564134528:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827331.295270:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827331.295280:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827331/real 1787827331] req@ffff976889cf92c0 x1874670564134528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827331.295288:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564134528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827331.295294:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827331.295297:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827331.295299:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564134528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827331.295303:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564134656:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827331.295314:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827331.295320:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827331/real 1787827331] req@ffff976889cfa1c0 x1874670564134656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827331.295327:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564134656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827331.295332:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827331.295334:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827331.295336:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564134656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827331.295340:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564134784:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827331.295357:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827331.295368:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827331/real 1787827331] req@ffff976889cfa580 x1874670564134784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827331.295388:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564134784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827331.295397:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827331.295400:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827331.295404:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564134784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827331.295410:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564134912:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827331.295426:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827331.295434:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827331/real 1787827331] req@ffff976889cfb480 x1874670564134912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827331.295441:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564134912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827331.295447:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827331.295449:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827331.295451:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564134912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827331.550952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827331.550956:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827331.551016:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827332.319001:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827332.574931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827332.574933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827332.574966:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827333.342924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827333.598926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827333.598926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827333.599014:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827334.366946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827334.622943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827334.622944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827334.622983:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827335.390913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827335.646930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827335.646962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827335.646998:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827336.414936:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827336.414938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827336.414946:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827336.414950:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827336.414958:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2276 00000100:00080000:3.0:1787827336.414967:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827336.414974:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827336.415010:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564135040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827336.415039:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827336.415042:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827336.415044:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827336.415048:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2276 00000100:00080000:3.0:1787827336.415052:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827336.415056:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827336.415064:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827336.415070:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564135040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827336.415080:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564135168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827336.415089:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827336.415092:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827336.415093:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827336.415096:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2276 00000100:00080000:3.0:1787827336.415099:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827336.415101:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827336.415104:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827336.415112:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564135296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827336.415125:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827336.415127:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787827336.415131:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827336.415131:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827336.415138:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2276 00000100:00080000:3.0:1787827336.415143:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827336.415146:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827336.415150:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827336.415159:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564135424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827336.415166:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827336.415169:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827336.415170:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827336.415173:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827336/real 1787827336] req@ffff97688cb5d2c0 x1874670564135040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827336.415176:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2276 00000100:00080000:3.0:1787827336.415183:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827336.415189:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827336.415193:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827336.415197:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564135040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827336.415207:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787827336.415211:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564135552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827336.415213:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827336.415221:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564135040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827336.415246:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564135168:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827336.415262:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827336.415271:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827336/real 1787827336] req@ffff97688cb5cf00 x1874670564135168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827336.415279:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564135168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827336.415285:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827336.415287:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827336.415290:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564135168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827336.415294:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564135296:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827336.415306:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827336.415312:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827336/real 1787827336] req@ffff97688cb5e1c0 x1874670564135296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827336.415320:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564135296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827336.415326:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827336.415327:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827336.415329:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564135296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827336.415335:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564135424:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827336.415354:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827336.415365:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827336/real 1787827336] req@ffff97688cb5c3c0 x1874670564135424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827336.415377:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564135424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827336.415386:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827336.415390:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827336.415394:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564135424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827336.415400:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564135552:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827336.415421:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827336.415444:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827336/real 1787827336] req@ffff97688cb5f840 x1874670564135552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827336.415461:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564135552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827336.415467:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827336.415469:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827336.415471:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564135552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827336.670936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827336.670936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827336.670969:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827337.439013:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827337.694945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827337.694991:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827337.695007:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827338.462937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827338.718938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827338.718959:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827338.718975:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827339.486924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827339.742904:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827339.742904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827339.742956:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827340.510930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827340.766924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827340.766928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827340.767002:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827341.534886:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827341.534894:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827341.534899:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827341.534902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827341.534907:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2281 00000100:00080000:3.0:1787827341.534918:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827341.534926:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827341.534972:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564135680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827341.535005:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827341.535009:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827341.535011:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827341.535017:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2281 00000100:00080000:3.0:1787827341.535021:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827341.535025:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827341.535040:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827341.535046:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564135680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827341.535055:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564135808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827341.535064:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827341.535067:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827341.535068:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827341.535071:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2281 00000100:00080000:3.0:1787827341.535074:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827341.535077:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827341.535080:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827341.535088:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564135936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827341.535095:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827341.535098:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827341.535099:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787827341.535105:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827341.535105:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2281 00000100:00080000:3.0:1787827341.535112:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827341.535115:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827341.535119:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827341.535127:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564136064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827341.535134:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827341.535136:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827341.535138:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827341.535141:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2281 00000100:00080000:3.0:1787827341.535144:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827341.535149:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827341.535151:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827341/real 1787827341] req@ffff97688cb5c780 x1874670564135680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827341.535152:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827341.535170:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564136192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827341.535178:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564135680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827341.535188:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827341.535192:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827341.535195:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564135680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827341.535216:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564135808:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827341.535255:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827341.535270:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827341/real 1787827341] req@ffff97688cb5cb40 x1874670564135808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827341.535284:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564135808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827341.535294:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827341.535297:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827341.535301:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564135808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827341.535308:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564135936:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827341.535327:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827341.535336:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827341/real 1787827341] req@ffff97688cb5c000 x1874670564135936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827341.535344:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564135936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827341.535349:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827341.535351:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827341.535354:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564135936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827341.535358:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564136064:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827341.535381:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827341.535388:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827341/real 1787827341] req@ffff97688cb5d680 x1874670564136064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827341.535401:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564136064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827341.535406:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827341.535408:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827341.535410:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564136064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827341.535413:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564136192:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827341.535423:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827341.535429:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827341/real 1787827341] req@ffff97688cb5f0c0 x1874670564136192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827341.535436:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564136192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827341.535441:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827341.535442:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827341.535444:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564136192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827341.790934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827341.790942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827341.790950:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827342.558954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827342.814971:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827342.814971:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827342.815017:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827343.583010:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827343.838922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827343.838969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827343.838995:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827344.606931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827344.862932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827344.862933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827344.863019:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827345.630925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827345.886926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827345.886933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827345.886964:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827346.654929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827346.654930:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827346.654941:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827346.654946:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827346.654959:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2286 00000100:00080000:3.0:1787827346.654973:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827346.654981:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827346.655029:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564136320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827346.655061:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827346.655067:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827346.655069:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827346.655073:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2286 00000100:00080000:3.0:1787827346.655076:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827346.655078:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827346.655090:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827346.655103:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564136448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827346.655113:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827346.655116:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827346.655118:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827346.655121:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2286 00000100:00080000:3.0:1787827346.655124:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827346.655126:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827346.655131:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827346.655134:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564136320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827346.655142:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564136576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827346.655152:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827346.655154:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827346.655155:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827346.655159:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2286 00000100:00080000:3.0:1787827346.655163:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827346.655166:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827346.655172:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827346.655188:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564136704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827346.655197:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827346.655200:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827346.655202:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827346.655205:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2286 00000100:00080000:3.0:1787827346.655215:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827346.655219:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827346.655223:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827346.655236:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564136832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827346.655238:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827346.655293:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827346/real 1787827346] req@ffff97688cb5de00 x1874670564136320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827346.655311:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564136320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827346.655322:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827346.655326:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827346.655329:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564136320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827346.655336:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564136448:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827346.655355:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827346.655366:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827346/real 1787827346] req@ffff97688cb5f0c0 x1874670564136448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827346.655376:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564136448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827346.655384:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827346.655386:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827346.655389:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564136448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827346.655421:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564136576:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827346.655435:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827346.655442:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827346/real 1787827346] req@ffff97688cb5d680 x1874670564136576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827346.655452:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564136576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827346.655464:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827346.655467:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827346.655470:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564136576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827346.655474:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564136704:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827346.655488:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827346.655495:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827346/real 1787827346] req@ffff97688cb5c000 x1874670564136704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827346.655503:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564136704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827346.655526:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827346.655528:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827346.655531:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564136704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827346.655534:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564136832:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827346.655547:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827346.655554:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827346/real 1787827346] req@ffff97688cb5cb40 x1874670564136832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827346.655562:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564136832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827346.655567:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827346.655569:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827346.655572:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564136832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827346.910995:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827346.910995:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827346.911102:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827347.678975:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827347.934932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827347.934932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827347.934997:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827348.702971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827348.958938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827348.958941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827348.959033:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827349.727036:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827349.982989:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827349.982989:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827349.983078:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827350.750922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827351.006895:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827351.006895:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827351.006940:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827351.774913:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827351.774918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827351.774925:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827351.774929:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827351.774937:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2291 00000100:00080000:3.0:1787827351.774953:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827351.774960:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827351.774992:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564136960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827351.775017:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827351.775020:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827351.775021:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827351.775024:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2291 00000100:00080000:3.0:1787827351.775027:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827351.775030:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827351.775038:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827351.775047:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564137088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827351.775054:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827351.775056:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827351.775058:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827351.775061:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2291 00000100:00080000:3.0:1787827351.775063:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827351.775065:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827351.775068:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827351.775072:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564136960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827351.775077:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564137216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827351.775086:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827351.775088:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827351.775089:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827351.775096:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2291 00000100:00080000:3.0:1787827351.775098:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827351.775101:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827351.775105:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827351.775115:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564137344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827351.775126:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827351.775128:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827351.775130:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827351.775132:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2291 00000100:00080000:3.0:1787827351.775135:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827351.775137:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827351.775140:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827351.775148:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827351.775151:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564137472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827351.775212:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827351/real 1787827351] req@ffff97688cb5c780 x1874670564136960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827351.775233:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564136960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827351.775245:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827351.775249:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827351.775254:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564136960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827351.775262:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564137088:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827351.775297:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827351.775311:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827351/real 1787827351] req@ffff97688cb5f840 x1874670564137088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827351.775325:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564137088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827351.775333:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827351.775337:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827351.775341:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564137088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827351.775368:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564137216:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827351.775392:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827351.775405:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827351/real 1787827351] req@ffff97688cb5c3c0 x1874670564137216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827351.775419:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564137216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827351.775429:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827351.775432:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827351.775436:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564137216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827351.775441:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564137344:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827351.775457:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827351.775464:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827351/real 1787827351] req@ffff97688cb5e1c0 x1874670564137344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827351.775472:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564137344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827351.775478:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827351.775480:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827351.775482:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564137344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827351.775486:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564137472:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827351.775495:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827351.775501:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827351/real 1787827351] req@ffff97688cb5cf00 x1874670564137472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827351.775511:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564137472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827351.775519:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827351.775522:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827351.775525:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564137472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827352.030927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827352.030927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827352.030978:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827352.798910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827353.054910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827353.054918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827353.054921:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827353.823026:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827354.078912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827354.078944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827354.078983:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827354.846950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827355.102906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827355.102932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827355.102976:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827355.870943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827356.126963:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827356.126963:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827356.127028:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827356.894901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827356.895898:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827356.895903:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827356.895907:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827356.895915:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2296 00000100:00080000:2.0:1787827356.895923:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827356.895928:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827356.895960:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564137600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827356.895987:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827356.895990:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827356.895991:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827356.895994:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2296 00000100:00080000:2.0:1787827356.895997:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827356.896001:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827356.896009:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827356.896017:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564137600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827356.896020:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564137728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827356.896030:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827356.896032:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827356.896034:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827356.896036:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2296 00000100:00080000:2.0:1787827356.896038:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827356.896040:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827356.896044:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827356.896051:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564137856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827356.896057:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827356.896059:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827356.896060:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827356.896064:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2296 00000100:00080000:2.0:1787827356.896066:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827356.896069:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827356.896073:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787827356.896074:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827356.896083:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564137984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827356.896091:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827356.896093:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827356.896095:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827356.896097:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2296 00000100:00080000:2.0:1787827356.896101:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827356.896104:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827356.896107:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827356.896109:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827356/real 1787827356] req@ffff976889cf9e00 x1874670564137600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827356.896117:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564138112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827356.896129:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564137600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827356.896138:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827356.896142:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827356.896145:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564137600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827356.896162:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564137728:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827356.896178:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827356.896186:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827356/real 1787827356] req@ffff976889cfa940 x1874670564137728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827356.896196:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564137728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827356.896202:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827356.896204:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827356.896206:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564137728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827356.896210:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564137856:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827356.896219:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827356.896225:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827356/real 1787827356] req@ffff976889cfbc00 x1874670564137856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827356.896233:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564137856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827356.896238:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827356.896240:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827356.896242:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564137856:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827356.896245:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564137984:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827356.896255:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827356.896261:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827356/real 1787827356] req@ffff976889cf8000 x1874670564137984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827356.896268:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564137984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827356.896277:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827356.896279:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827356.896282:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564137984:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827356.896285:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564138112:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827356.896294:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827356.896300:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827356/real 1787827356] req@ffff976889cfb0c0 x1874670564138112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827356.896306:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564138112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827356.896311:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827356.896313:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827356.896314:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564138112:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827357.150898:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827357.150984:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827357.151029:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827357.918933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827358.174981:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827358.175010:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827358.175056:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827358.942915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827359.198938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827359.198939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827359.199008:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827359.966902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827360.222893:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827360.222893:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827360.222942:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827360.990912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827361.246895:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827361.246896:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827361.246916:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827362.014940:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827362.014947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827362.014948:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827362.014954:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827362.014964:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2301 00000100:00080000:2.0:1787827362.014974:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827362.014980:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827362.015015:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564138240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827362.015038:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827362.015041:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827362.015044:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827362.015050:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2301 00000100:00080000:2.0:1787827362.015056:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827362.015060:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827362.015070:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827362.015076:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564138240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827362.015090:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564138368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827362.015106:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827362.015111:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827362.015114:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827362.015119:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2301 00000100:00080000:2.0:1787827362.015125:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827362.015129:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827362.015134:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787827362.015140:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827362.015149:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564138496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827362.015158:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827362.015162:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827362.015164:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827362.015172:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2301 00000100:00080000:2.0:1787827362.015176:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827362.015179:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827362.015185:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827362.015197:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564138624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827362.015204:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827362/real 1787827362] req@ffff976889cf8f00 x1874670564138240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827362.015208:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827362.015213:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827362.015215:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827362.015223:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2301 00000100:00080000:2.0:1787827362.015228:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827362.015231:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827362.015236:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827362.015250:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564138752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827362.015254:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564138240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827362.015280:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827362.015286:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827362.015291:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564138240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827362.015317:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564138368:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827362.015346:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827362.015365:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827362/real 1787827362] req@ffff976889cfb0c0 x1874670564138368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827362.015379:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564138368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827362.015388:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827362.015393:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827362.015396:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564138368:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827362.015403:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564138496:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827362.015426:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827362.015442:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827362/real 1787827362] req@ffff976889cf8000 x1874670564138496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827362.015454:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564138496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827362.015463:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827362.015467:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827362.015471:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564138496:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827362.015480:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564138624:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827362.015510:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827362.015525:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827362/real 1787827362] req@ffff976889cfbc00 x1874670564138624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827362.015539:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564138624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827362.015549:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827362.015553:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827362.015557:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564138624:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827362.015565:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564138752:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827362.015589:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827362.015606:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827362/real 1787827362] req@ffff976889cfa940 x1874670564138752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827362.015623:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564138752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827362.015636:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827362.015641:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827362.015647:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564138752:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827362.270913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827362.270940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827362.270953:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827363.038897:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827363.294894:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827363.294900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827363.294961:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827364.062912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:0.0:1787827364.318947:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827364.318972:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827364.318976:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827365.086899:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827365.342892:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827365.342908:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827365.342947:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827366.110914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827366.366918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827366.366920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827366.366985:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827367.134997:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827367.135003:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827367.135010:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827367.135016:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827367.135026:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2307 00000100:00080000:0.0:1787827367.135045:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827367.135053:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827367.135095:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564138880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827367.135140:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827367.135143:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827367.135146:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827367.135152:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2307 00000100:00080000:0.0:1787827367.135156:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827367.135161:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827367.135175:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827367.135206:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564138880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827367.135211:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564139008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827367.135224:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827367.135228:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827367.135229:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827367.135234:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2307 00000100:00080000:0.0:1787827367.135237:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827367.135241:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827367.135246:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827367.135260:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564139136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827367.135269:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827367.135272:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827367.135274:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827367.135278:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2307 00000100:00080000:0.0:1787827367.135285:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827367.135306:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827367.135311:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787827367.135318:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827367.135337:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564139264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827367.135349:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827367.135352:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827367.135353:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827367.135357:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2307 00000100:00080000:0.0:1787827367.135360:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827367.135364:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827367.135368:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827367.135378:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827367/real 1787827367] req@ffff976882aed680 x1874670564138880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827367.135381:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564139392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827367.135424:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564138880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827367.135439:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827367.135444:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827367.135449:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564138880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827367.135480:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564139008:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827367.135525:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827367.135541:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827367/real 1787827367] req@ffff976882aec3c0 x1874670564139008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827367.135557:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564139008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827367.135585:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827367.135589:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827367.135594:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564139008:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827367.135602:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564139136:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827367.135627:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827367.135656:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827367/real 1787827367] req@ffff976882aeed00 x1874670564139136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827367.135674:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564139136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827367.135685:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827367.135689:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827367.135694:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564139136:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827367.135702:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564139264:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827367.135728:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827367.135743:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827367/real 1787827367] req@ffff976882aee580 x1874670564139264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827367.135760:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564139264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827367.135771:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827367.135775:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827367.135779:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564139264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827367.135806:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564139392:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827367.135873:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827367.135887:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827367/real 1787827367] req@ffff976882aee940 x1874670564139392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827367.135902:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564139392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827367.135911:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827367.135915:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827367.135918:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564139392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827367.390915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827367.390915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827367.390959:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827368.158902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827368.414925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827368.414967:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827368.415109:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827369.182938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827369.438940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827369.438953:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827369.439054:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827370.206899:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827370.462901:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827370.462921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827370.462936:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827371.230967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827371.486967:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827371.486985:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827371.487028:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827372.254979:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827372.255002:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827372.255011:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827372.255016:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827372.255027:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2312 00000100:00080000:3.0:1787827372.255040:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827372.255060:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827372.255114:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564139520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827372.255146:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827372.255150:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827372.255152:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827372.255157:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2312 00000100:00080000:3.0:1787827372.255161:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827372.255165:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827372.255176:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827372.255191:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564139648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827372.255198:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564139520:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827372.255236:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827372.255241:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827372.255243:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827372.255248:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2312 00000100:00080000:3.0:1787827372.255253:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827372.255257:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827372.255263:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827372.255274:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564139776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827372.255293:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827372.255297:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1787827372.255303:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827372.255303:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827372.255311:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2312 00000100:00080000:3.0:1787827372.255317:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827372.255321:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827372.255327:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827372.255339:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564139904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827372.255349:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827372.255353:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827372.255357:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827372.255359:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827372/real 1787827372] req@ffff97688cb5d2c0 x1874670564139520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827372.255366:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2312 00000100:00080000:3.0:1787827372.255376:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827372.255385:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827372.255390:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827372.255396:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564139520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827372.255409:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564140032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827372.255414:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827372.255427:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827372.255437:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564139520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827372.255476:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564139648:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827372.255499:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827372.255512:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827372/real 1787827372] req@ffff97688cb5cf00 x1874670564139648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827372.255544:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564139648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827372.255554:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827372.255558:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827372.255562:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564139648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827372.255569:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564139776:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827372.255590:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827372.255601:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827372/real 1787827372] req@ffff97688cb5e1c0 x1874670564139776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827372.255671:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564139776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827372.255685:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827372.255690:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827372.255696:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564139776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827372.255706:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564139904:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827372.255740:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827372.255756:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827372/real 1787827372] req@ffff97688cb5c3c0 x1874670564139904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827372.255770:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564139904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827372.255778:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827372.255782:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827372.255785:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564139904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827372.255791:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564140032:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827372.255810:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827372.255820:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827372/real 1787827372] req@ffff97688cb5f840 x1874670564140032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827372.255831:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564140032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827372.255874:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827372.255877:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827372.255881:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564140032:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827372.510947:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827372.510970:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827372.511089:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827373.279017:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827373.535008:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827373.535013:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827373.535069:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827374.302910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827374.558985:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827374.558988:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827374.559108:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827375.326959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827375.582961:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827375.582961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827375.583061:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827376.350988:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827376.606896:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827376.606897:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827376.607000:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827377.374911:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827377.374915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827377.374923:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827377.374928:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827377.374938:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2317 00000100:00080000:3.0:1787827377.374949:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827377.374957:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827377.374995:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564140160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827377.375019:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827377.375023:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827377.375025:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827377.375028:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2317 00000100:00080000:3.0:1787827377.375031:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827377.375033:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827377.375041:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827377.375050:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564140288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827377.375059:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827377.375061:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827377.375062:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827377.375068:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564140160:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827377.375069:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2317 00000100:00080000:3.0:1787827377.375074:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827377.375078:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827377.375081:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827377.375088:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564140416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827377.375095:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827377.375097:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827377.375099:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827377.375102:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2317 00000100:00080000:3.0:1787827377.375104:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827377.375106:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827377.375110:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827377.375117:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564140544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827377.375122:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827377.375125:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827377.375126:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827377.375129:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2317 00000100:00080000:3.0:1787827377.375131:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827377.375133:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827377.375136:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827377.375144:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564140672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827377.375168:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564140288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827377.375192:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564140416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827377.375208:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564140544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827377.375220:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564140672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827377.630923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827377.630923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827377.631002:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827378.398981:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827378.654928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827378.654929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827378.654983:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827379.423029:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827379.678912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827379.678912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827379.678976:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827380.447031:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:0.0:1787827380.447119:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:0.0:1787827380.447172:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827380.447189:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787827380.447230:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787827380.447241:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787827380.447249:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787827380.447257:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:3.0:1787827380.447263:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827377/real 1787827380] req@ffff97688cb5cb40 x1874670564140288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827380.447296:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564140288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827380.447310:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827380.447315:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827380.447320:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564140288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827380.447328:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827377/real 1787827380] req@ffff97688cb5c000 x1874670564140416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827380.447342:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564140416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827380.447353:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827380.447357:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827380.447361:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564140416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827380.447368:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827377/real 1787827380] req@ffff97688cb5d680 x1874670564140544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827380.447382:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564140544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827380.447391:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827380.447395:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827380.447398:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564140544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827380.447413:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827377/real 1787827380] req@ffff97688cb5f0c0 x1874670564140672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827380.447426:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564140672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827380.447435:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827380.447439:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827380.447442:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564140672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827380.447450:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827377/real 1787827380] req@ffff97688cb5c780 x1874670564140160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827380.447463:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564140160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827380.447471:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827380.447474:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827380.447478:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564140160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827380.702978:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827380.702978:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827380.703036:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827381.470967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827381.726928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827381.726928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827381.727004:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827382.494876:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827382.494900:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827382.494904:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827382.494912:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2322 00000100:00080000:3.0:1787827382.494920:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827382.494926:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827382.494957:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564140800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827382.494982:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827382.494984:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827382.494989:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827382.494991:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827382.494995:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2322 00000100:00080000:3.0:1787827382.494998:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827382.495002:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827382.495010:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827382.495018:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564140928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827382.495025:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827382.495027:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827382.495029:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827382.495031:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2322 00000100:00080000:3.0:1787827382.495033:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827382.495035:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827382.495038:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827382.495043:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564141056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827382.495050:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827382.495052:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827382.495053:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827382.495056:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2322 00000100:00080000:3.0:1787827382.495058:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827382.495061:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827382.495064:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827382.495071:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564140800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827382.495072:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564141184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827382.495084:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827382.495086:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827382.495087:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827382.495090:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2322 00000100:00080000:3.0:1787827382.495093:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827382.495096:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827382.495099:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827382.495107:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564141312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827382.495140:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827382.495179:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827382/real 1787827382] req@ffff97688cb5c780 x1874670564140800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827382.495193:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564140800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827382.495202:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827382.495204:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827382.495207:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564140800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827382.495213:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564140928:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827382.495231:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827382.495240:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827382/real 1787827382] req@ffff97688cb5f0c0 x1874670564140928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827382.495248:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564140928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827382.495253:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827382.495256:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827382.495259:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564140928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827382.495265:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564141056:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827382.495284:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827382.495295:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827382/real 1787827382] req@ffff97688cb5d680 x1874670564141056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827382.495308:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564141056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827382.495318:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827382.495321:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827382.495325:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564141056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827382.495363:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564141184:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827382.495387:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827382.495400:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827382/real 1787827382] req@ffff97688cb5c000 x1874670564141184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827382.495418:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564141184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827382.495426:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827382.495429:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827382.495432:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564141184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827382.495439:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564141312:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827382.495460:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827382.495472:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827382/real 1787827382] req@ffff97688cb5cb40 x1874670564141312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827382.495485:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564141312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827382.495494:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827382.495497:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827382.495501:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564141312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827382.750920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827382.750955:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827382.750999:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827383.519002:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827383.774956:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827383.774975:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827383.774994:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827384.543030:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827384.799013:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787827384.799024:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827384.799073:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827385.566952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787827385.822982:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827385.822994:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827385.823036:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827386.590970:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827386.846942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827386.847008:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827386.847050:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827387.614951:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827387.614956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827387.614970:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827387.614977:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827387.614990:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2327 00000100:00080000:3.0:1787827387.615004:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827387.615012:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827387.615058:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564141440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827387.615108:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827387.615112:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827387.615114:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827387.615118:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2327 00000100:00080000:3.0:1787827387.615121:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827387.615125:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827387.615138:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827387.615153:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564141568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827387.615164:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827387.615172:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564141440:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827387.615178:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827387.615183:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827387.615189:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2327 00000100:00080000:3.0:1787827387.615192:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827387.615195:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827387.615201:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827387.615213:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564141696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827387.615222:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827387.615226:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827387.615227:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827387.615232:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2327 00000100:00080000:3.0:1787827387.615236:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827387.615239:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827387.615242:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827387.615253:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564141824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827387.615261:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787827387.615267:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827387.615269:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827387.615274:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827387.615279:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2327 00000100:00080000:3.0:1787827387.615282:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827387.615285:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827387.615289:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827387.615301:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564141952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827387.615342:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827387/real 1787827387] req@ffff97688cb5de00 x1874670564141440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827387.615387:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564141440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827387.615398:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827387.615402:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827387.615406:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564141440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827387.615465:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564141568:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827387.615489:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827387.615530:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827387/real 1787827387] req@ffff97688cb5cb40 x1874670564141568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827387.615542:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564141568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827387.615549:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827387.615552:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827387.615555:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564141568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827387.615560:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564141696:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827387.615590:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827387.615599:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827387/real 1787827387] req@ffff97688cb5c000 x1874670564141696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827387.615609:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564141696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827387.615622:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827387.615625:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827387.615628:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564141696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827387.615633:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564141824:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827387.615646:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827387.615653:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827387/real 1787827387] req@ffff97688cb5d680 x1874670564141824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827387.615663:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564141824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827387.615669:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827387.615672:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827387.615674:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564141824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827387.615679:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564141952:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827387.615692:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827387.615699:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827387/real 1787827387] req@ffff97688cb5f0c0 x1874670564141952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827387.615709:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564141952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827387.615715:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827387.615717:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827387.615720:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564141952:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827387.870915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827387.870915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827387.870941:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827388.639004:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827388.894914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827388.894920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827388.895025:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827389.662930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827389.918885:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827389.918888:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827389.918916:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827390.687066:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827390.942971:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827390.943035:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827390.943100:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827391.710954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827391.966973:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827391.966984:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827391.967006:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827392.734947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827392.734948:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827392.734957:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827392.734961:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827392.734969:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2332 00000100:00080000:0.0:1787827392.734981:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827392.734988:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827392.735024:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564142080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827392.735073:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827392.735076:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827392.735077:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827392.735084:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2332 00000100:00080000:0.0:1787827392.735088:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827392.735092:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827392.735101:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827392.735109:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564142080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827392.735115:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564142208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827392.735126:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827392.735128:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827392.735131:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827392.735139:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2332 00000100:00080000:0.0:1787827392.735143:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827392.735146:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827392.735150:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827392.735159:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564142336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827392.735167:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827392.735169:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827392.735171:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787827392.735176:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787827392.735179:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2332 00000100:00080000:0.0:1787827392.735184:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827392.735187:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827392.735191:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827392.735201:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564142464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827392.735208:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827392.735210:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827392.735212:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827392.735228:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2332 00000100:00080000:0.0:1787827392.735231:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827392.735235:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827392/real 1787827392] req@ffff976882aef0c0 x1874670564142080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827392.735237:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827392.735266:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827392.735303:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564142592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827392.735327:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564142080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827392.735347:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827392.735352:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827392.735358:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564142080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827392.735416:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564142208:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827392.735458:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827392.735494:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827392/real 1787827392] req@ffff976882aee940 x1874670564142208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827392.735505:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564142208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827392.735512:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827392.735515:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827392.735517:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564142208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827392.735530:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564142336:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827392.735549:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827392.735557:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827392/real 1787827392] req@ffff976882aee580 x1874670564142336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827392.735566:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564142336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827392.735573:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827392.735575:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827392.735578:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564142336:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827392.735582:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564142464:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827392.735594:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827392.735607:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827392/real 1787827392] req@ffff976882aeed00 x1874670564142464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827392.735615:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564142464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827392.735621:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827392.735623:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827392.735626:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564142464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827392.735629:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564142592:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827392.735641:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827392.735647:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827392/real 1787827392] req@ffff976882aec3c0 x1874670564142592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827392.735657:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564142592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827392.735662:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827392.735664:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827392.735667:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564142592:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827392.990933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827392.990934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827392.990995:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827393.758958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787827394.014950:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827394.014952:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827394.014957:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827394.783069:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827395.038944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827395.038951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827395.039131:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827395.806908:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827396.062915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827396.062915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827396.062954:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827396.830926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827397.086879:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827397.086906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827397.087026:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827397.854900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827397.855062:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827397.855075:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827397.855082:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827397.855114:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2337 00000100:00080000:3.0:1787827397.855129:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827397.855141:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827397.855212:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564142720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827397.855280:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827397.855289:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827397.855294:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827397.855305:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2337 00000100:00080000:3.0:1787827397.855316:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827397.855326:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827397.855346:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827397.855349:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564142720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827397.855403:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564142848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827397.855453:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827397.855488:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827397.855492:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827397.855500:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2337 00000100:00080000:3.0:1787827397.855508:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827397.855526:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827397.855581:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827397/real 1787827397] req@ffff97688cb5c780 x1874670564142720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827397.855609:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564142720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827397.855623:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827397.855628:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827397.855634:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564142720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827397.855664:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564142848:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827397.855704:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827397.855719:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827397/real 1787827397] req@ffff97688cb5f840 x1874670564142848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827397.855730:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564142848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827397.855741:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827397.855745:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827397.855747:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564142848:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787827397.855807:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827397.855947:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827397.855988:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564142976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827397.856029:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827397.856038:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827397.856043:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827397.856056:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564142976:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787827397.856057:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2337 00000100:00080000:3.0:1787827397.856077:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827397.856086:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:1.0:1787827397.856097:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:3.0:1787827397.856098:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827397.856117:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827397/real 1787827397] req@ffff97688cb5c3c0 x1874670564142976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827397.856126:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564143104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827397.856145:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564142976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827397.856209:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827397.856210:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827397.856227:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827397.856229:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00080000:3.0:1787827397.856232:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827397.856239:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564142976:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787827397.856248:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2337 00000100:00100000:1.0:1787827397.856266:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564143104:192.168.203.139@tcp:8:kworker.0 00000100:00080000:3.0:1787827397.856281:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827397.856293:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:1.0:1787827397.856301:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:3.0:1787827397.856304:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827397.856314:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827397/real 1787827397] req@ffff97688cb5e1c0 x1874670564143104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827397.856324:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564143104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827397.856331:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827397.856338:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827397.856339:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564143232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827397.856344:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564143104:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827397.856446:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564143232:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827397.856496:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827397.856505:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827397/real 1787827397] req@ffff97688cb5cf00 x1874670564143232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827397.856514:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564143232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827397.856520:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827397.856523:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827397.856526:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564143232:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827398.110906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827398.110948:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827398.110967:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827398.878923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827399.134934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827399.134949:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827399.135005:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827399.902956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827400.158919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827400.158940:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827400.158944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827400.926933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827401.182937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827401.182939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827401.183058:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827401.950942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827402.206918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827402.206919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827402.206956:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827402.974941:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827402.974951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827402.974954:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827402.974962:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827402.974976:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2342 00000100:00080000:3.0:1787827402.974990:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827402.975017:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827402.975065:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564143360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827402.975112:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827402.975116:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827402.975118:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827402.975123:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2342 00000100:00080000:3.0:1787827402.975128:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827402.975135:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827402.975151:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827402.975165:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564143360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827402.975180:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564143488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827402.975212:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827402.975217:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827402.975219:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827402.975225:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2342 00000100:00080000:3.0:1787827402.975228:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827402.975232:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827402.975237:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827402.975247:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564143616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827402.975255:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827402.975258:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827402.975259:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827402.975263:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2342 00000100:00080000:3.0:1787827402.975267:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827402.975272:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827402.975274:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827402.975281:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827402.975290:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564143744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827402.975297:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827402.975300:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827402.975317:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827402.975322:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2342 00000100:00080000:3.0:1787827402.975327:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827402.975332:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827402.975338:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827402.975347:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827402/real 1787827402] req@ffff97688cb5d2c0 x1874670564143360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827402.975353:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564143872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827402.975442:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564143360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827402.975454:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827402.975458:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827402.975478:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564143360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827402.975510:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564143488:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827402.975538:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827402.975552:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827402/real 1787827402] req@ffff97688cb5cf00 x1874670564143488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827402.975564:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564143488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827402.975573:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827402.975577:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827402.975580:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564143488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827402.975585:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564143616:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827402.975601:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827402.975610:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827402/real 1787827402] req@ffff97688cb5e1c0 x1874670564143616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827402.975621:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564143616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827402.975636:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827402.975639:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827402.975642:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564143616:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827402.975647:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564143744:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827402.975663:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827402.975688:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827402/real 1787827402] req@ffff97688cb5c3c0 x1874670564143744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827402.975696:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564143744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827402.975704:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827402.975706:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827402.975709:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564143744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827402.975713:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564143872:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827402.975727:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827402.975751:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827402/real 1787827402] req@ffff97688cb5f840 x1874670564143872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827402.975769:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564143872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827402.975781:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827402.975786:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827402.975791:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564143872:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827403.230960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827403.230976:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827403.231066:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827403.998965:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827404.254960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827404.254967:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827404.254984:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827405.022903:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827405.278919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827405.278919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827405.279020:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827406.046924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827406.302906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827406.302913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827406.302959:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827407.070934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827407.326936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827407.326936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827407.326991:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827408.094912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827408.094974:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827408.094990:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827408.094994:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827408.095003:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2347 00000100:00080000:0.0:1787827408.095013:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827408.095019:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827408.095058:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564144000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827408.095083:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827408.095086:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827408.095088:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827408.095092:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2347 00000100:00080000:0.0:1787827408.095096:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827408.095100:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827408.095110:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827408.095133:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564144128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827408.095144:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564144000:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787827408.095145:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827408.095153:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827408.095159:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827408.095162:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2347 00000100:00080000:0.0:1787827408.095165:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827408.095167:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827408.095171:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827408.095182:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564144256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827408.095189:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827408.095192:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827408.095193:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827408.095197:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2347 00000100:00080000:0.0:1787827408.095236:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827408.095241:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827408.095249:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1787827408.095258:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827408.095274:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564144384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827408.095314:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827408.095317:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827408.095319:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827408.095323:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2347 00000100:00080000:0.0:1787827408.095326:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827408.095329:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827408.095335:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827408.095345:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827408/real 1787827408] req@ffff976882aed680 x1874670564144000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827408.095349:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564144512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827408.095422:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564144000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827408.095433:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827408.095437:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827408.095441:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564144000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827408.095464:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564144128:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827408.095488:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827408.095504:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827408/real 1787827408] req@ffff976882aec3c0 x1874670564144128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827408.095520:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564144128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827408.095533:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827408.095537:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827408.095542:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564144128:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827408.095550:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564144256:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827408.095572:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827408.095582:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827408/real 1787827408] req@ffff976882aeed00 x1874670564144256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827408.095592:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564144256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827408.095599:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827408.095601:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827408.095603:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564144256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827408.095608:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564144384:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827408.095621:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827408.095628:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827408/real 1787827408] req@ffff976882aee580 x1874670564144384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827408.095637:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564144384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827408.095643:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827408.095645:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827408.095648:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564144384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827408.095652:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564144512:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827408.095663:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827408.095670:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827408/real 1787827408] req@ffff976882aee940 x1874670564144512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827408.095678:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564144512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827408.095684:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827408.095686:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827408.095689:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564144512:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827408.350991:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827408.350992:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827408.351030:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827409.118907:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827409.374983:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827409.375017:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827409.375078:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827410.143015:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827410.399027:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827410.399034:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827410.399061:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827411.167022:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827411.422923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827411.422924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827411.422987:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827412.190928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827412.446928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827412.446928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827412.446954:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827413.214970:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827413.215019:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827413.215045:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827413.215052:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827413.215067:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2353 00000100:00080000:2.0:1787827413.215081:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827413.215092:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827413.215140:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564144640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827413.215181:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827413.215202:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827413.215205:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827413.215213:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2353 00000100:00080000:2.0:1787827413.215219:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827413.215225:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827413.215237:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564144640:192.168.203.139@tcp:250:kworker.0 00000008:00100000:2.0:1787827413.215239:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827413.215262:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564144768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827413.215289:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827413.215294:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827413.215297:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827413.215302:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2353 00000100:00080000:2.0:1787827413.215309:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827413.215314:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827413.215348:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827413.215382:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827413.215390:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564144896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827413.215407:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827413.215412:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827413.215414:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827413.215421:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2353 00000100:00080000:2.0:1787827413.215428:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827413.215434:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827413.215444:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827413/real 1787827413] req@ffff976889cf9e00 x1874670564144640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827413.215446:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827413.215473:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564145024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827413.215506:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827413.215514:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787827413.215515:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564144640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827413.215518:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827413.215534:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2353 00000100:00080000:2.0:1787827413.215545:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827413.215547:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827413.215555:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:1.0:1787827413.215562:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000008:00100000:2.0:1787827413.215567:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827413.215572:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564144640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827413.215587:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564145152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827413.215613:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564144768:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827413.215655:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827413.215680:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827413/real 1787827413] req@ffff976889cfb480 x1874670564144768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827413.215706:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564144768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827413.215725:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827413.215731:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827413.215739:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564144768:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827413.215751:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564144896:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827413.215780:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827413.215801:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827413/real 1787827413] req@ffff976889cfa580 x1874670564144896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827413.215816:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564144896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827413.215877:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827413.215881:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827413.215885:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564144896:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827413.215893:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564145024:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827413.215917:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827413.215930:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827413/real 1787827413] req@ffff976889cfa1c0 x1874670564145024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827413.215944:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564145024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827413.215954:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827413.215958:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827413.215961:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564145024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827413.215968:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564145152:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827413.215985:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827413.215996:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827413/real 1787827413] req@ffff976889cf92c0 x1874670564145152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827413.216010:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564145152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827413.216030:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827413.216033:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827413.216038:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564145152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827413.470887:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827413.470895:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827413.470942:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827414.238970:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827414.494892:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827414.494912:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827414.494926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827415.262983:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827415.518979:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827415.518997:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827415.519014:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827416.286892:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827416.542941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827416.542948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827416.542979:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827417.310915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827417.566976:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827417.566977:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827417.567042:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827418.334949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827418.334975:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827418.334983:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827418.334989:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827418.335001:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2358 00000100:00080000:0.0:1787827418.335014:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827418.335023:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827418.335081:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564145280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827418.335125:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827418.335129:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827418.335132:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827418.335138:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2358 00000100:00080000:0.0:1787827418.335145:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827418.335151:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827418.335164:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827418.335173:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564145280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827418.335201:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564145408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827418.335216:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827418.335220:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827418.335223:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827418.335228:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2358 00000100:00080000:0.0:1787827418.335253:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827418.335257:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827418.335264:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827418.335267:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827418.335304:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564145536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827418.335322:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827418.335326:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827418.335329:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827418.335336:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2358 00000100:00080000:0.0:1787827418.335342:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827418.335351:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827418.335353:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827418/real 1787827418] req@ffff976882aef0c0 x1874670564145280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827418.335359:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827418.335385:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564145664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827418.335391:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564145280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827418.335434:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827418.335437:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827418.335439:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827418.335447:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827418.335450:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:0.0:1787827418.335456:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2358 00000100:00080000:0.0:1787827418.335464:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827418.335464:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564145280:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787827418.335474:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827418.335482:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827418.335493:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564145792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827418.335506:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564145408:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827418.335541:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827418.335557:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827418/real 1787827418] req@ffff976882aee940 x1874670564145408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827418.335571:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564145408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827418.335581:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827418.335585:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827418.335589:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564145408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827418.335595:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564145536:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827418.335613:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827418.335623:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827418/real 1787827418] req@ffff976882aee580 x1874670564145536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827418.335634:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564145536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827418.335643:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827418.335646:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827418.335649:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564145536:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827418.335655:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564145664:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827418.335671:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827418.335680:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827418/real 1787827418] req@ffff976882aeed00 x1874670564145664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827418.335692:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564145664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827418.335700:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827418.335703:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827418.335707:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564145664:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827418.335722:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564145792:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827418.335745:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827418.335761:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827418/real 1787827418] req@ffff976882aec3c0 x1874670564145792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827418.335781:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564145792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827418.335796:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827418.335802:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827418.335808:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564145792:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827418.590944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827418.590947:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827418.590975:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827419.359077:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827419.614965:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827419.615046:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827419.615142:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827420.382962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827420.638961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827420.638961:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827420.639007:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827421.406964:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827421.662959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827421.662979:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827421.663083:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827422.430937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827422.686921:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827422.686934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827422.686959:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827423.454913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827423.454916:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827423.454927:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827423.454933:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827423.454946:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2363 00000100:00080000:0.0:1787827423.454959:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827423.454968:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827423.455017:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564145920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827423.455046:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827423.455050:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827423.455051:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827423.455056:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2363 00000100:00080000:0.0:1787827423.455061:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827423.455066:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827423.455079:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827423.455086:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564145920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827423.455106:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564146048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827423.455121:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827423.455125:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827423.455129:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827423.455132:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2363 00000100:00080000:0.0:1787827423.455151:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827423.455155:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827423.455160:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827423.455163:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827423.455178:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564146176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827423.455186:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827423.455189:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827423.455191:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827423.455194:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2363 00000100:00080000:0.0:1787827423.455197:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827423.455201:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827423.455206:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827423.455210:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827423/real 1787827423] req@ffff976882aed680 x1874670564145920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827423.455219:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564146304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827423.455264:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827423.455267:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827423.455269:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827423.455274:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2363 00000100:00100000:1.0:1787827423.455274:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564145920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827423.455279:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827423.455286:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827423.455293:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827423.455295:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827423.455303:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827423.455305:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564146432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827423.455322:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564145920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827423.455353:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564146048:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827423.455374:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827423.455393:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827423/real 1787827423] req@ffff976882aec3c0 x1874670564146048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827423.455403:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564146048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827423.455409:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827423.455411:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827423.455414:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564146048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827423.455418:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564146176:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827423.455431:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827423.455438:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827423/real 1787827423] req@ffff976882aeed00 x1874670564146176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827423.455445:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564146176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827423.455456:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827423.455458:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827423.455460:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564146176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827423.455464:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564146304:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827423.455475:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827423.455481:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827423/real 1787827423] req@ffff976882aee580 x1874670564146304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827423.455488:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564146304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827423.455494:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827423.455496:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827423.455497:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564146304:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827423.455501:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564146432:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827423.455511:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827423.455516:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827423/real 1787827423] req@ffff976882aee940 x1874670564146432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827423.455524:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564146432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827423.455536:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827423.455541:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827423.455544:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564146432:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827423.710941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827423.710977:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827423.711005:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827424.478919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827424.734962:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827424.734985:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827424.735026:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827425.502971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787827425.758877:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827425.758939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827425.759015:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827426.526963:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827426.782932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827426.783001:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827426.783025:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827427.550910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827427.806886:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827427.806902:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827427.806959:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827428.574897:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827428.574914:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827428.574922:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827428.574927:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827428.574937:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2368 00000100:00080000:3.0:1787827428.574948:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827428.574964:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827428.574997:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564146560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827428.575039:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827428.575041:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827428.575043:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827428.575046:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2368 00000100:00080000:3.0:1787827428.575049:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827428.575053:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827428.575061:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827428.575074:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564146560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827428.575074:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564146688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827428.575090:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827428.575092:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827428.575093:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827428.575095:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2368 00000100:00080000:3.0:1787827428.575097:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827428.575100:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827428.575103:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827428.575111:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564146816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827428.575129:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827428.575131:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827428.575132:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827428.575135:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2368 00000100:00080000:3.0:1787827428.575138:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827428.575141:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827428.575144:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827428.575152:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564146944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827428.575158:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827428.575164:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827428.575168:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827428.575170:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827428.575175:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2368 00000100:00080000:3.0:1787827428.575179:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827428.575184:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827428.575190:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827428.575196:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827428/real 1787827428] req@ffff97688cb5c780 x1874670564146560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827428.575206:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564147072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827428.575238:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564146560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827428.575248:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827428.575251:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827428.575255:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564146560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827428.575275:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564146688:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827428.575294:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827428.575303:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827428/real 1787827428] req@ffff97688cb5f0c0 x1874670564146688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827428.575311:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564146688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827428.575318:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827428.575320:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827428.575322:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564146688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827428.575326:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564146816:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827428.575337:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827428.575342:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827428/real 1787827428] req@ffff97688cb5d680 x1874670564146816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827428.575349:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564146816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827428.575355:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827428.575357:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827428.575358:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564146816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827428.575362:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564146944:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827428.575372:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827428.575384:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827428/real 1787827428] req@ffff97688cb5c000 x1874670564146944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827428.575391:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564146944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827428.575396:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827428.575398:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827428.575400:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564146944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827428.575404:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564147072:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827428.575414:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827428.575419:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827428/real 1787827428] req@ffff97688cb5cb40 x1874670564147072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827428.575425:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564147072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827428.575431:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827428.575433:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827428.575435:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564147072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827428.830893:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827428.830918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827428.830940:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827429.598979:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827429.854878:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827429.854897:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827429.854953:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827430.622892:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827430.878904:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827430.878909:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827430.878934:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827431.646968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827431.902948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827431.902987:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827431.903072:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827432.670912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827432.926912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827432.926926:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827432.926954:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787827433.694981:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827433.695025:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827433.695062:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827433.695068:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827433.695083:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2373 00000100:00080000:2.0:1787827433.695099:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827433.695109:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827433.695159:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564147200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827433.695243:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827433.695259:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827433.695261:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827433.695268:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2373 00000100:00080000:2.0:1787827433.695276:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827433.695281:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827433.695296:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827433.695301:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564147200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827433.695324:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564147328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827433.695345:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827433.695349:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827433.695352:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827433.695359:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2373 00000100:00080000:2.0:1787827433.695364:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827433.695373:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827433.695379:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827433.695405:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564147456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827433.695416:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827433.695421:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827433.695423:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827433.695427:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2373 00000100:00080000:2.0:1787827433.695431:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827433.695435:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827433.695440:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827433.695451:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564147584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827433.695462:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827433.695465:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827433.695467:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827433.695471:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2373 00000100:00080000:2.0:1787827433.695474:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827433.695478:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827433.695483:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827433.695493:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564147712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827433.695646:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827433.695720:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827433/real 1787827433] req@ffff976889cfad00 x1874670564147200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827433.695748:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564147200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827433.695818:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827433.695845:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827433.695852:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564147200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827433.695875:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564147328:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827433.695904:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827433.695919:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827433/real 1787827433] req@ffff976889cf92c0 x1874670564147328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827433.695929:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564147328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827433.695935:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827433.695938:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827433.695940:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564147328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827433.695945:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564147456:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827433.695957:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827433.695965:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827433/real 1787827433] req@ffff976889cfa1c0 x1874670564147456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827433.695977:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564147456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827433.696001:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827433.696006:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827433.696011:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564147456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827433.696021:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564147584:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827433.696049:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827433.696074:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827433/real 1787827433] req@ffff976889cfa580 x1874670564147584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827433.696083:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564147584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827433.696089:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827433.696091:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827433.696093:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564147584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827433.696098:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564147712:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827433.696110:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827433.696116:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827433/real 1787827433] req@ffff976889cfb480 x1874670564147712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827433.696123:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564147712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827433.696128:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827433.696130:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827433.696132:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564147712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827433.950925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827433.950937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827433.951010:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827434.718933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827434.974911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827434.974915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827434.974925:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827435.743061:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827435.998892:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827435.998910:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827435.999004:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827436.766900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827437.022920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827437.022920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827437.023015:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827437.791013:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827438.046952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827438.046973:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827438.047051:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827438.814947:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827438.814953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827438.814958:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827438.814965:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827438.814977:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2378 00000100:00080000:3.0:1787827438.814998:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827438.815008:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827438.815055:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564147840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827438.815083:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827438.815086:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827438.815088:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827438.815091:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2378 00000100:00080000:3.0:1787827438.815095:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827438.815098:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827438.815109:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827438.815121:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564147840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827438.815123:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564147968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827438.815134:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827438.815136:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827438.815138:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827438.815156:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2378 00000100:00080000:3.0:1787827438.815162:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827438.815166:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827438.815170:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827438.815181:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564148096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827438.815189:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827438.815192:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827438.815193:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827438.815197:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2378 00000100:00080000:3.0:1787827438.815199:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827438.815203:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827438.815206:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827438.815217:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564148224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827438.815218:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827438.815232:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827438.815235:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827438.815239:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827438.815242:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2378 00000100:00080000:3.0:1787827438.815245:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827438.815248:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827438.815251:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827438.815261:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564148352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827438.815265:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827438/real 1787827438] req@ffff97688cb5de00 x1874670564147840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827438.815288:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564147840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827438.815298:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827438.815313:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827438.815317:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564147840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827438.815350:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564147968:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827438.815368:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827438.815378:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827438/real 1787827438] req@ffff97688cb5cb40 x1874670564147968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827438.815386:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564147968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827438.815393:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827438.815406:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827438.815409:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564147968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827438.815413:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564148096:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827438.815424:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827438.815431:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827438/real 1787827438] req@ffff97688cb5c000 x1874670564148096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827438.815444:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564148096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827438.815452:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827438.815456:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827438.815459:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564148096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827438.815465:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564148224:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827438.815488:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827438.815499:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827438/real 1787827438] req@ffff97688cb5d680 x1874670564148224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827438.815507:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564148224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827438.815512:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827438.815515:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827438.815528:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564148224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827438.815532:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564148352:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827438.815544:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827438.815550:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827438/real 1787827438] req@ffff97688cb5f0c0 x1874670564148352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827438.815557:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564148352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827438.815563:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827438.815565:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827438.815567:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564148352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827439.070906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827439.070911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827439.070945:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827439.838956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827440.094929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827440.094929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827440.094978:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827440.862971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827441.118908:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827441.118939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827441.118948:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827441.886937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827442.142909:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827442.142912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827442.142964:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827442.910946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827443.166918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827443.166940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827443.166950:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827443.934945:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827443.934951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827443.934954:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827443.934961:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827443.934972:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2383 00000100:00080000:3.0:1787827443.934983:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827443.934992:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827443.935045:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564148480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827443.935077:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827443.935081:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827443.935083:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827443.935089:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2383 00000100:00080000:3.0:1787827443.935094:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827443.935100:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827443.935125:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564148480:192.168.203.139@tcp:250:kworker.0 00000008:00100000:3.0:1787827443.935125:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827443.935160:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564148608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827443.935179:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827443.935182:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827443.935184:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827443.935186:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2383 00000100:00080000:3.0:1787827443.935190:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827443.935193:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827443.935197:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827443.935204:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564148736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827443.935216:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827443.935218:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827443.935220:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827443.935222:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2383 00000100:00080000:3.0:1787827443.935225:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827443.935227:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827443.935231:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827443.935239:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564148864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827443.935245:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827443.935249:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827443.935250:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827443.935255:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564148608:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787827443.935257:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2383 00000100:00080000:3.0:1787827443.935263:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827443.935266:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827443.935270:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827443.935281:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564148992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827443.935291:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564148736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827443.935311:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564148864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827443.935329:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564148992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827444.190935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827444.190989:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827444.191028:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827444.958922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827445.214900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827445.214914:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827445.214915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827445.982958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827446.238959:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827446.238959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827446.239041:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827447.006952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:2.0:1787827447.007231:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:2.0:1787827447.007278:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787827447.007299:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787827447.007345:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787827447.007355:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787827447.007367:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787827447.007381:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:1.0:1787827447.007417:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827443/real 1787827447] req@ffff97688cb5f840 x1874670564148608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827447.007454:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564148608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827447.007481:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827447.007486:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827447.007491:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564148608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827447.007502:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827443/real 1787827447] req@ffff97688cb5c3c0 x1874670564148736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827447.007516:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564148736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827447.007523:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827447.007526:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827447.007530:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564148736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827447.007539:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827443/real 1787827447] req@ffff97688cb5e1c0 x1874670564148864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827447.007556:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564148864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827447.007567:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827447.007575:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827447.007579:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564148864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827447.007592:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827443/real 1787827447] req@ffff97688cb5c780 x1874670564148480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827447.007608:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564148480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827447.007619:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827447.007623:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827447.007627:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564148480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827447.007676:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827443/real 1787827447] req@ffff97688cb5cf00 x1874670564148992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827447.007693:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564148992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827447.007702:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827447.007707:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827447.007710:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564148992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827447.262963:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827447.262963:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827447.262997:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827448.030943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827448.286932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827448.287010:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827448.287045:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827449.054959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827449.055014:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827449.055019:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827449.055023:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827449.055034:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2388 00000100:00080000:1.0:1787827449.055045:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827449.055055:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827449.055110:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564149120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827449.055154:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827449.055157:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827449.055158:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827449.055162:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2388 00000100:00080000:1.0:1787827449.055165:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827449.055169:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827449.055179:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827449.055190:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564149248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827449.055194:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564149120:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827449.055200:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827449.055205:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827449.055207:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827449.055210:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2388 00000100:00080000:1.0:1787827449.055212:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827449.055214:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827449.055217:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827449.055226:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564149376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827449.055232:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827449.055235:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827449.055237:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827449.055240:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2388 00000100:00080000:1.0:1787827449.055243:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827449.055246:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827449.055249:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827449.055257:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564149504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827449.055262:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827449.055265:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827449.055267:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827449.055269:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2388 00000100:00080000:1.0:1787827449.055274:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827449.055276:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827449.055279:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827449.055292:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564149632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827449.055292:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827449.055351:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827449/real 1787827449] req@ffff976889efcb40 x1874670564149120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827449.055374:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564149120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827449.055398:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827449.055404:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827449.055408:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564149120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827449.055440:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564149248:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827449.055490:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827449.055506:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827449/real 1787827449] req@ffff976889efe1c0 x1874670564149248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827449.055526:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564149248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827449.055536:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827449.055539:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827449.055543:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564149248:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827449.055549:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564149376:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827449.055572:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827449.055584:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827449/real 1787827449] req@ffff976889efe580 x1874670564149376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827449.055596:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564149376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827449.055611:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827449.055615:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827449.055618:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564149376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827449.055625:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564149504:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827449.055646:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827449.055656:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827449/real 1787827449] req@ffff976889eff480 x1874670564149504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827449.055668:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564149504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827449.055677:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827449.055680:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827449.055683:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564149504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827449.055689:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564149632:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827449.055707:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827449.055718:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827449/real 1787827449] req@ffff976889efc780 x1874670564149632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827449.055729:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564149632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827449.055737:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827449.055740:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827449.056249:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564149632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827449.310930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827449.310930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827449.310958:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827450.078903:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827450.334911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827450.334932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827450.334967:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827451.102913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827451.358917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827451.358930:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827451.359011:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827452.126908:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827452.382912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827452.382952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827452.382986:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827453.150948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827453.407024:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827453.407027:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827453.407100:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827454.174996:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827454.175001:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827454.175010:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827454.175014:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827454.175025:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2394 00000100:00080000:2.0:1787827454.175071:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827454.175080:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827454.175149:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564149760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827454.175177:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827454.175181:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827454.175182:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827454.175186:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2394 00000100:00080000:2.0:1787827454.175189:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827454.175191:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827454.175218:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827454.175231:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564149888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827454.175240:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827454.175244:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827454.175246:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827454.175248:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2394 00000100:00080000:2.0:1787827454.175251:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827454.175255:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827454.175260:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564149760:192.168.203.139@tcp:250:kworker.0 00000008:00100000:2.0:1787827454.175261:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827454.175274:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564150016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827454.175283:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827454.175286:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827454.175288:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827454.175293:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2394 00000100:00080000:2.0:1787827454.175296:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827454.175299:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827454.175303:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827454.175314:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564150144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827454.175320:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827454.175324:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827454.175325:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827454.175329:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2394 00000100:00080000:2.0:1787827454.175333:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827454.175336:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:3.0:1787827454.175346:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:2.0:1787827454.175348:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827454.175361:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564150272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827454.175399:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827454/real 1787827454] req@ffff976889cf9e00 x1874670564149760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827454.175415:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564149760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827454.175423:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827454.175427:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827454.175430:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564149760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827454.175436:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564149888:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827454.175452:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827454.175461:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827454/real 1787827454] req@ffff976889cfa940 x1874670564149888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827454.175469:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564149888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827454.175480:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827454.175482:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827454.175485:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564149888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827454.175507:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564150016:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827454.175520:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827454.175526:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827454/real 1787827454] req@ffff976889cfbc00 x1874670564150016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827454.175534:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564150016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827454.175539:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827454.175541:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827454.175544:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564150016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827454.175547:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564150144:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827454.175557:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827454.175563:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827454/real 1787827454] req@ffff976889cf8000 x1874670564150144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827454.175570:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564150144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827454.175576:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827454.175578:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827454.175579:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564150144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827454.175583:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564150272:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827454.175592:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827454.175598:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827454/real 1787827454] req@ffff976889cfb0c0 x1874670564150272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827454.175613:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564150272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827454.175622:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827454.175625:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827454.175629:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564150272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827454.430992:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827454.430993:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827454.431057:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827455.198943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827455.454933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827455.454950:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827455.454953:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827456.222906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827456.478915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827456.478949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827456.478966:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827457.246920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827457.502897:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827457.502901:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827457.502908:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827458.270906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827458.526894:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827458.526902:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827458.527002:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827459.294932:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827459.294935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827459.294945:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827459.294950:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827459.294960:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2399 00000100:00080000:2.0:1787827459.294971:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827459.294977:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827459.295025:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564150400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827459.295048:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827459.295051:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827459.295052:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827459.295055:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2399 00000100:00080000:2.0:1787827459.295058:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827459.295060:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827459.295068:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827459.295080:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564150528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827459.295091:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827459.295094:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827459.295096:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827459.295107:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2399 00000100:00100000:3.0:1787827459.295109:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564150400:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787827459.295114:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827459.295125:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827459.295129:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827459.295138:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564150656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827459.295147:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827459.295150:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827459.295152:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827459.295156:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2399 00000100:00080000:2.0:1787827459.295159:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827459.295162:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827459.295166:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827459.295175:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564150784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827459.295181:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827459.295183:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827459.295185:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827459.295188:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2399 00000100:00080000:2.0:1787827459.295192:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827459.295195:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827459.295200:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787827459.295205:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827459.295218:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564150912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827459.295262:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827459/real 1787827459] req@ffff976889cf8f00 x1874670564150400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827459.295279:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564150400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827459.295289:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827459.295292:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827459.295295:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564150400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827459.295328:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564150528:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827459.295346:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827459.295356:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827459/real 1787827459] req@ffff976889cfb0c0 x1874670564150528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827459.295364:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564150528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827459.295371:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827459.295374:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827459.295376:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564150528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827459.295380:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564150656:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827459.295392:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827459.295397:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827459/real 1787827459] req@ffff976889cf8000 x1874670564150656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827459.295405:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564150656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827459.295410:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827459.295412:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827459.295414:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564150656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827459.295418:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564150784:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827459.295429:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827459.295435:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827459/real 1787827459] req@ffff976889cfbc00 x1874670564150784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827459.295441:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564150784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827459.295446:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827459.295448:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827459.295451:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564150784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827459.295454:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564150912:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827459.295464:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827459.295470:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827459/real 1787827459] req@ffff976889cfa940 x1874670564150912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827459.295476:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564150912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827459.295481:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827459.295482:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827459.295484:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564150912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827459.550930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827459.550937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827459.550964:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827460.318949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827460.574937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827460.574937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827460.575001:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827461.342921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827461.598948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827461.598948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827461.598999:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827462.367000:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827462.622909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827462.622909:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827462.622963:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827463.390889:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827463.646915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827463.646966:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827463.646966:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827464.414992:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827464.414998:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827464.415007:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827464.415012:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827464.415024:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2404 00000100:00080000:3.0:1787827464.415037:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827464.415046:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827464.415093:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564151040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827464.415118:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827464.415121:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827464.415123:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827464.415126:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2404 00000100:00080000:3.0:1787827464.415129:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827464.415132:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827464.415142:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827464.415154:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564151168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827464.415160:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564151040:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827464.415164:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827464.415169:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827464.415171:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827464.415174:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2404 00000100:00080000:3.0:1787827464.415176:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827464.415186:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827464.415190:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827464.415198:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564151296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827464.415205:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827464.415208:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827464.415209:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827464.415212:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2404 00000100:00080000:3.0:1787827464.415215:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827464.415217:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827464.415221:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827464.415229:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564151424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827464.415236:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827464.415238:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827464.415240:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787827464.415245:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827464.415246:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2404 00000100:00080000:3.0:1787827464.415251:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827464.415254:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827464.415256:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827464.415266:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564151552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827464.415307:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827464/real 1787827464] req@ffff97688cb5d2c0 x1874670564151040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827464.415327:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564151040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827464.415339:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827464.415344:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827464.415349:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564151040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827464.415377:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564151168:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827464.415399:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827464.415411:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827464/real 1787827464] req@ffff97688cb5cf00 x1874670564151168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827464.415425:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564151168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827464.415434:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827464.415438:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827464.415441:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564151168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827464.415447:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564151296:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827464.415464:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827464.415474:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827464/real 1787827464] req@ffff97688cb5c780 x1874670564151296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827464.415485:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564151296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827464.415494:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827464.415498:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827464.415502:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564151296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827464.415507:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564151424:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827464.415524:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827464.415533:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827464/real 1787827464] req@ffff97688cb5e1c0 x1874670564151424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827464.415546:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564151424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827464.415555:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827464.415558:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827464.415562:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564151424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827464.415567:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564151552:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827464.415583:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827464.415592:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827464/real 1787827464] req@ffff97688cb5c3c0 x1874670564151552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827464.415604:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564151552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827464.415612:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827464.415616:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827464.415619:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564151552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827464.670890:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827464.670893:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827464.670992:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827465.438911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827465.694912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827465.694924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827465.694968:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827466.462940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827466.719012:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827466.719012:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827466.719128:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827467.486903:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827467.742896:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827467.742919:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827467.742953:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827468.510930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827468.766963:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827468.766963:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827468.767054:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827469.534888:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827469.534897:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827469.534901:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827469.534910:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2409 00000100:00080000:3.0:1787827469.534920:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827469.534940:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827469.534966:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787827469.534977:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564151680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827469.535005:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827469.535008:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827469.535011:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827469.535016:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2409 00000100:00080000:3.0:1787827469.535020:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827469.535024:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827469.535032:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827469.535043:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564151808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827469.535051:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827469.535053:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827469.535055:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827469.535058:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2409 00000100:00080000:3.0:1787827469.535060:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827469.535063:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827469.535066:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827469.535075:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564151936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827469.535086:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827469.535089:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827469.535092:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827469.535097:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2409 00000100:00080000:3.0:1787827469.535101:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827469.535107:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827469.535113:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827469.535131:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564151680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827469.535133:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564152064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827469.535154:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827469.535158:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827469.535162:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827469.535168:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2409 00000100:00080000:3.0:1787827469.535176:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827469.535182:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827469.535188:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827469.535202:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827469.535205:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564152192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827469.535248:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827469/real 1787827469] req@ffff97688cb5f840 x1874670564151680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827469.535268:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564151680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827469.535280:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827469.535284:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827469.535290:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564151680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827469.535300:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564151808:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827469.535326:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827469.535341:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827469/real 1787827469] req@ffff97688cb5f0c0 x1874670564151808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827469.535355:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564151808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827469.535365:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827469.535368:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827469.535372:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564151808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827469.535378:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564151936:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827469.535398:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827469.535410:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827469/real 1787827469] req@ffff97688cb5d680 x1874670564151936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827469.535430:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564151936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827469.535438:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827469.535442:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827469.535446:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564151936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827469.535480:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564152064:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827469.535499:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827469.535516:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827469/real 1787827469] req@ffff97688cb5c000 x1874670564152064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827469.535524:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564152064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827469.535531:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827469.535534:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827469.535536:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564152064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827469.535542:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564152192:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827469.535557:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827469.535569:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827469/real 1787827469] req@ffff97688cb5cb40 x1874670564152192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827469.535580:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564152192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827469.535593:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827469.535597:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827469.535601:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564152192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827469.790898:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827469.790904:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827469.790956:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827470.558963:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827470.814929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827470.814930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827470.814994:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827471.582881:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827471.838946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827471.838967:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827471.838985:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827472.606949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827472.862890:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827472.862929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827472.862972:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827473.630931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827473.886880:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827473.886922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827473.886939:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827474.654929:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827474.654937:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1787827474.654939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827474.654940:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827474.654952:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2414 00000100:00080000:2.0:1787827474.654960:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827474.654967:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827474.654999:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564152320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827474.655019:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827474.655022:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827474.655024:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827474.655027:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2414 00000100:00080000:2.0:1787827474.655029:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827474.655032:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827474.655040:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827474.655054:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564152448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827474.655062:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827474.655066:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827474.655069:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827474.655071:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564152320:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787827474.655073:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2414 00000100:00080000:2.0:1787827474.655079:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827474.655083:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827474.655086:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827474.655093:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564152576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827474.655100:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827474.655102:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827474.655103:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827474.655107:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2414 00000100:00080000:2.0:1787827474.655109:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827474.655112:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827474.655115:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827474.655122:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564152704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827474.655128:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827474.655131:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827474.655132:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827474.655134:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2414 00000100:00080000:2.0:1787827474.655137:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827474.655142:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787827474.655142:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827474.655145:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827474.655154:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564152832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827474.655182:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827474/real 1787827474] req@ffff976889cf9e00 x1874670564152320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827474.655204:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564152320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827474.655215:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827474.655219:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827474.655223:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564152320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827474.655247:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564152448:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827474.655268:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827474.655278:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827474/real 1787827474] req@ffff976889cfb480 x1874670564152448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827474.655287:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564152448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827474.655294:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827474.655297:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827474.655301:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564152448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827474.655307:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564152576:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827474.655326:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827474.655338:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827474/real 1787827474] req@ffff976889cfa580 x1874670564152576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827474.655350:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564152576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827474.655356:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827474.655358:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827474.655360:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564152576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827474.655364:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564152704:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827474.655378:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827474.655385:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827474/real 1787827474] req@ffff976889cfa1c0 x1874670564152704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827474.655393:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564152704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827474.655398:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827474.655400:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827474.655404:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564152704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827474.655410:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564152832:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827474.655428:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827474.655439:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827474/real 1787827474] req@ffff976889cf92c0 x1874670564152832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827474.655452:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564152832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827474.655460:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827474.655462:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827474.655464:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564152832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827474.910900:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827474.910978:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827474.910990:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827475.678902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827475.934901:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827475.934951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827475.934958:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827476.702923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827476.958942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827476.958950:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827476.958977:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827477.726924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827477.982918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827477.982937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827477.982970:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827478.750980:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827479.006917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827479.006967:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827479.007027:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827479.774938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827479.774954:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827479.774962:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827479.774968:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827479.774979:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2419 00000100:00080000:2.0:1787827479.774991:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827479.774998:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827479.775044:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564152960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827479.775074:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827479.775077:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827479.775079:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827479.775083:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2419 00000100:00080000:2.0:1787827479.775085:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827479.775088:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827479.775098:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827479.775110:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564153088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827479.775118:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827479.775121:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827479.775122:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827479.775125:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2419 00000100:00080000:2.0:1787827479.775127:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827479.775129:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827479.775133:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827479.775136:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564152960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827479.775142:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564153216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827479.775151:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827479.775153:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827479.775155:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827479.775158:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2419 00000100:00080000:2.0:1787827479.775160:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827479.775162:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827479.775166:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827479.775174:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564153344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827479.775180:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827479.775183:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827479.775184:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827479.775187:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2419 00000100:00080000:2.0:1787827479.775189:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827479.775191:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827479.775194:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827479.775201:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564153472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827479.775214:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827479.775260:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827479/real 1787827479] req@ffff976889cfad00 x1874670564152960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827479.775275:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564152960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827479.775283:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827479.775287:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827479.775291:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564152960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827479.775297:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564153088:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827479.775314:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827479.775328:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827479/real 1787827479] req@ffff976889cf92c0 x1874670564153088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827479.775342:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564153088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827479.775352:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827479.775356:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827479.775359:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564153088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827479.775397:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564153216:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827479.775415:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827479.775423:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827479/real 1787827479] req@ffff976889cfa1c0 x1874670564153216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827479.775431:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564153216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827479.775438:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827479.775440:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827479.775442:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564153216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827479.775446:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564153344:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827479.775457:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827479.775464:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827479/real 1787827479] req@ffff976889cfa580 x1874670564153344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827479.775471:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564153344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827479.775476:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827479.775478:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827479.775480:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564153344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827479.775483:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564153472:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827479.775494:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827479.775499:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827479/real 1787827479] req@ffff976889cfb480 x1874670564153472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827479.775507:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564153472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827479.775512:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827479.775514:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827479.775515:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564153472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827480.030925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827480.030932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827480.031013:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827480.798955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827481.054923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827481.054953:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827481.054956:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827481.822929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827482.078912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827482.078914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827482.078947:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827482.846898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827483.102893:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827483.102912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827483.102956:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827483.870950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827484.126904:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827484.126905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827484.126941:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827484.894961:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827484.894964:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827484.894974:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827484.894980:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827484.894991:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2424 00000100:00080000:0.0:1787827484.895019:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827484.895025:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827484.895059:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564153600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827484.895080:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827484.895084:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827484.895086:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827484.895090:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2424 00000100:00080000:0.0:1787827484.895094:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827484.895099:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827484.895109:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827484.895116:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564153600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827484.895122:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564153728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827484.895133:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827484.895136:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827484.895137:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827484.895141:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2424 00000100:00080000:0.0:1787827484.895144:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827484.895148:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827484.895152:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827484.895161:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564153856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827484.895170:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827484.895173:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827484.895175:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827484.895178:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2424 00000100:00080000:0.0:1787827484.895181:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827484.895183:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787827484.895185:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827484.895189:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827484.895198:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564153984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827484.895207:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827484.895210:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827484.895212:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827484.895215:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2424 00000100:00080000:0.0:1787827484.895218:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827484.895221:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827484/real 1787827484] req@ffff976882aef0c0 x1874670564153600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827484.895222:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827484.895226:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827484.895235:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564154112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827484.895247:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564153600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827484.895258:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827484.895262:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827484.895266:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564153600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827484.895290:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564153728:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827484.895316:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827484.895330:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827484/real 1787827484] req@ffff976882aee940 x1874670564153728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827484.895344:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564153728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827484.895352:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827484.895355:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827484.895357:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564153728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827484.895361:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564153856:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827484.895377:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827484.895385:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827484/real 1787827484] req@ffff976882aee580 x1874670564153856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827484.895397:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564153856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827484.895402:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827484.895404:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827484.895406:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564153856:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827484.895410:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564153984:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827484.895420:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827484.895426:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827484/real 1787827484] req@ffff976882aeed00 x1874670564153984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827484.895433:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564153984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827484.895438:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827484.895440:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827484.895442:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564153984:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827484.895445:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564154112:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827484.895454:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827484.895459:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827484/real 1787827484] req@ffff976882aec3c0 x1874670564154112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827484.895466:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564154112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827484.895476:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827484.895478:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827484.895480:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564154112:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827485.150916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827485.150917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827485.150946:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827485.918931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827486.174969:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827486.174970:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827486.175069:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827486.942890:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827487.198932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827487.198932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827487.199002:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827487.966980:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827488.222944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827488.222944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827488.223031:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827488.990925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827489.246913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827489.246914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827489.246953:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827490.014911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827490.014915:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827490.014922:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827490.014926:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827490.014935:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2429 00000100:00080000:0.0:1787827490.014950:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827490.014961:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827490.015000:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564154240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827490.015025:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827490.015029:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827490.015031:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827490.015052:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2429 00000100:00080000:0.0:1787827490.015058:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827490.015063:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827490.015074:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827490.015080:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564154240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827490.015096:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564154368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827490.015110:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827490.015115:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827490.015117:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827490.015122:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2429 00000100:00080000:0.0:1787827490.015129:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827490.015134:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:1.0:1787827490.015139:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:0.0:1787827490.015142:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827490.015157:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564154496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827490.015165:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827490.015168:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827490.015169:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827490.015172:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2429 00000100:00080000:0.0:1787827490.015176:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827490.015179:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827490.015183:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827490.015185:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827490/real 1787827490] req@ffff976882aed680 x1874670564154240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827490.015195:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564154624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827490.015207:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564154240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827490.015211:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827490.015234:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827490.015237:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827490.015244:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2429 00000100:00080000:1.0:1787827490.015246:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827490.015247:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827490.015255:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:1.0:1787827490.015255:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000008:00100000:0.0:1787827490.015263:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827490.015267:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564154240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827490.015274:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564154752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827490.015301:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564154368:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827490.015326:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827490.015345:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827490/real 1787827490] req@ffff976882aec3c0 x1874670564154368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827490.015354:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564154368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827490.015361:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827490.015364:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827490.015366:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564154368:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827490.015370:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564154496:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827490.015383:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827490.015390:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827490/real 1787827490] req@ffff976882aeed00 x1874670564154496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827490.015404:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564154496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827490.015410:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827490.015413:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827490.015415:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564154496:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827490.015418:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564154624:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827490.015430:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827490.015461:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827490/real 1787827490] req@ffff976882aee580 x1874670564154624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827490.015483:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564154624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827490.015495:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827490.015500:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827490.015520:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564154624:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827490.015526:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564154752:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827490.015547:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827490.015557:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827490/real 1787827490] req@ffff976882aee940 x1874670564154752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827490.015565:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564154752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827490.015598:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827490.015601:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827490.015604:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564154752:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827490.270962:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827490.270963:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827490.271060:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827491.038946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827491.294914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827491.294919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827491.294937:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827492.062898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827492.318925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827492.318933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827492.318957:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827493.086971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827493.342882:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827493.342893:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827493.342915:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827494.111046:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827494.366908:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827494.366960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827494.366966:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827495.134908:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827495.134921:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827495.134937:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827495.134942:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827495.134953:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2435 00000100:00080000:3.0:1787827495.134966:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827495.134976:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827495.135035:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564154880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827495.135065:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827495.135068:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827495.135069:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827495.135073:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2435 00000100:00080000:3.0:1787827495.135076:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827495.135091:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827495.135102:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827495.135116:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564154880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827495.135118:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564155008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827495.135129:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827495.135132:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827495.135134:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827495.135137:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2435 00000100:00080000:3.0:1787827495.135140:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827495.135142:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827495.135146:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827495.135154:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564155136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827495.135161:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827495.135163:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827495.135165:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827495.135167:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2435 00000100:00080000:3.0:1787827495.135169:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827495.135171:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827495.135181:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827495.135192:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564155264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827495.135199:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787827495.135204:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827495.135205:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827495.135209:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827495.135213:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2435 00000100:00080000:3.0:1787827495.135216:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827495.135219:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827495.135222:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827495.135231:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564155392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827495.135261:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827495/real 1787827495] req@ffff97688cb5de00 x1874670564154880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827495.135276:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564154880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827495.135286:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827495.135290:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827495.135294:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564154880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827495.135347:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564155008:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827495.135375:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827495.135389:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827495/real 1787827495] req@ffff97688cb5cb40 x1874670564155008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827495.135413:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564155008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827495.135423:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827495.135427:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827495.135430:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564155008:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827495.135437:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564155136:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827495.135458:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827495.135468:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827495/real 1787827495] req@ffff97688cb5c000 x1874670564155136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827495.135476:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564155136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827495.135483:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827495.135485:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827495.135487:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564155136:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827495.135491:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564155264:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827495.135505:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827495.135512:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827495/real 1787827495] req@ffff97688cb5d680 x1874670564155264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827495.135520:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564155264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827495.135525:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827495.135528:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827495.135531:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564155264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827495.135536:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564155392:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827495.135556:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827495.135568:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827495/real 1787827495] req@ffff97688cb5f0c0 x1874670564155392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827495.135580:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564155392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827495.135586:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827495.135589:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827495.135591:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564155392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827495.390968:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827495.390973:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827495.391008:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827496.158927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827496.414921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827496.414975:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827496.414977:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827497.182948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787827497.438888:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827497.438917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827497.438950:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827498.206892:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827498.462977:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827498.463013:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827498.463082:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827499.230967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827499.486928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827499.486955:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827499.486966:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827500.254938:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827500.254939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827500.254950:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827500.254956:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827500.254978:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2440 00000100:00080000:3.0:1787827500.254991:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827500.254999:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827500.255045:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564155520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827500.255071:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827500.255074:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827500.255076:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827500.255079:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2440 00000100:00080000:3.0:1787827500.255082:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827500.255084:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827500.255094:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827500.255104:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564155648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827500.255113:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827500.255116:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787827500.255125:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564155520:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827500.255125:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827500.255133:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2440 00000100:00080000:3.0:1787827500.255136:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827500.255139:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827500.255143:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827500.255152:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564155776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827500.255159:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827500.255162:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827500.255164:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827500.255167:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2440 00000100:00080000:3.0:1787827500.255169:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827500.255172:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827500.255175:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827500.255182:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564155904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827500.255189:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827500.255191:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827500.255192:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827500.255195:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2440 00000100:00080000:3.0:1787827500.255197:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827500.255200:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:1.0:1787827500.255204:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:3.0:1787827500.255207:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827500.255218:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564156032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827500.255271:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827500/real 1787827500] req@ffff97688cb5f840 x1874670564155520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827500.255292:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564155520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827500.255303:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827500.255306:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827500.255309:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564155520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827500.255315:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564155648:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827500.255336:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827500.255346:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827500/real 1787827500] req@ffff97688cb5c3c0 x1874670564155648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827500.255355:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564155648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827500.255361:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827500.255364:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827500.255366:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564155648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827500.255394:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564155776:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827500.255406:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827500.255412:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827500/real 1787827500] req@ffff97688cb5e1c0 x1874670564155776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827500.255420:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564155776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827500.255431:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827500.255433:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827500.255436:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564155776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827500.255439:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564155904:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827500.255449:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827500.255454:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827500/real 1787827500] req@ffff97688cb5c780 x1874670564155904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827500.255462:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564155904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827500.255467:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827500.255469:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827500.255471:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564155904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827500.255474:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564156032:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827500.255485:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827500.255490:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827500/real 1787827500] req@ffff97688cb5cf00 x1874670564156032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827500.255497:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564156032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827500.255502:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827500.255504:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827500.255507:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564156032:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827500.510926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827500.511038:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827500.511052:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827501.278995:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827501.534916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827501.535009:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827501.535016:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827502.302939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827502.558968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827502.558970:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827502.559116:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827503.326964:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827503.582977:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827503.582992:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827503.583040:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827504.350908:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827504.606914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827504.606937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827504.606984:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827505.374985:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827505.374994:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827505.375000:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827505.375007:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827505.375020:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2445 00000100:00080000:3.0:1787827505.375035:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827505.375044:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827505.375090:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564156160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827505.375121:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827505.375124:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827505.375125:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827505.375129:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2445 00000100:00080000:3.0:1787827505.375132:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827505.375134:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827505.375144:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827505.375156:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564156288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827505.375164:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827505.375167:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827505.375170:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827505.375171:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564156160:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827505.375174:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2445 00000100:00080000:3.0:1787827505.375179:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827505.375183:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827505.375196:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827505.375210:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564156416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827505.375220:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827505.375224:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827505.375226:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827505.375231:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2445 00000100:00080000:3.0:1787827505.375236:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827505.375241:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:1.0:1787827505.375245:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:3.0:1787827505.375249:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827505.375262:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564156544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827505.375270:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827505.375272:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827505.375274:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827505.375277:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2445 00000100:00080000:3.0:1787827505.375280:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827505.375283:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827505.375287:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827505.375291:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827505/real 1787827505] req@ffff97688cb5d2c0 x1874670564156160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827505.375297:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564156672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827505.375316:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564156160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827505.375335:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827505.375339:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827505.375343:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564156160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827505.375368:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564156288:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827505.375393:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827505.375406:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827505/real 1787827505] req@ffff97688cb5cf00 x1874670564156288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827505.375415:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564156288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827505.375422:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827505.375425:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827505.375428:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564156288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827505.375433:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564156416:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827505.375447:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827505.375454:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827505/real 1787827505] req@ffff97688cb5c780 x1874670564156416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827505.375463:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564156416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827505.375469:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827505.375472:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827505.375474:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564156416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827505.375478:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564156544:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827505.375494:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827505.375516:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827505/real 1787827505] req@ffff97688cb5e1c0 x1874670564156544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827505.375530:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564156544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827505.375539:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827505.375542:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827505.375547:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564156544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827505.375556:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564156672:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827505.375580:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827505.375591:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827505/real 1787827505] req@ffff97688cb5c3c0 x1874670564156672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827505.375604:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564156672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827505.375610:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827505.375612:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827505.375614:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564156672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827505.630967:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827505.630967:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827505.630995:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827506.398921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827506.654928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827506.654928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827506.654960:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827507.422936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827507.678940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827507.678940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827507.678964:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827508.446894:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827508.702914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827508.702916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827508.703023:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827509.470925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827509.726907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827509.726907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827509.726937:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827510.494925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827510.494935:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827510.494943:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827510.494949:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827510.494960:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2450 00000100:00080000:3.0:1787827510.494972:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827510.494979:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827510.495031:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564156800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827510.495063:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827510.495066:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827510.495068:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827510.495072:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2450 00000100:00080000:3.0:1787827510.495074:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827510.495077:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827510.495086:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827510.495097:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564156928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827510.495104:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827510.495107:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827510.495108:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827510.495111:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2450 00000100:00080000:3.0:1787827510.495113:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827510.495118:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827510.495119:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564156800:192.168.203.139@tcp:250:kworker.0 00000008:00100000:3.0:1787827510.495123:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827510.495131:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564157056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827510.495137:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827510.495140:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827510.495141:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827510.495143:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2450 00000100:00080000:3.0:1787827510.495145:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827510.495148:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827510.495151:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827510.495159:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564157184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827510.495165:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827510.495168:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827510.495169:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827510.495172:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2450 00000100:00080000:3.0:1787827510.495174:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827510.495176:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827510.495179:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827510.495188:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564157312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827510.495241:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564156928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827510.495294:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564157056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827510.495324:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564157184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827510.495352:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564157312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827510.750900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827510.750974:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827510.751002:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827511.518960:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827511.774950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827511.774950:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827511.774985:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827512.542988:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827512.798898:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827512.798898:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827512.798952:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827513.566987:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:1.0:1787827513.567187:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:1.0:1787827513.567245:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787827513.567269:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827513.567340:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827513.567358:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827513.567373:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827513.567390:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:2.0:1787827513.567397:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827510/real 1787827513] req@ffff97688cb5f840 x1874670564156800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827513.567445:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564156800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827513.567460:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827513.567466:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827513.567470:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564156800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827513.567484:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827510/real 1787827513] req@ffff97688cb5f0c0 x1874670564156928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827513.567498:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564156928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827513.567507:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827513.567511:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827513.567515:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564156928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827513.567545:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827510/real 1787827513] req@ffff97688cb5d680 x1874670564157056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827513.567558:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564157056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827513.567574:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827513.567578:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827513.567581:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564157056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827513.567590:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827510/real 1787827513] req@ffff97688cb5c000 x1874670564157184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827513.567604:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564157184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827513.567613:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827513.567617:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827513.567637:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564157184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827513.567644:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827510/real 1787827513] req@ffff97688cb5cb40 x1874670564157312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827513.567657:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564157312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827513.567666:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827513.567670:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827513.567673:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564157312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827513.822912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827513.822931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827513.822958:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827514.590950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827514.846951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827514.846966:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827514.847107:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827515.614984:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827515.614988:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827515.615005:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827515.615011:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827515.615021:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2455 00000100:00080000:0.0:1787827515.615032:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827515.615040:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827515.615081:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564157440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827515.615100:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827515.615103:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827515.615105:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827515.615109:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2455 00000100:00080000:0.0:1787827515.615112:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827515.615115:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827515.615124:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827515.615134:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564157568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827515.615143:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827515.615145:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827515.615148:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827515.615153:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2455 00000100:00080000:0.0:1787827515.615156:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827515.615161:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827515.615164:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564157440:192.168.203.139@tcp:250:kworker.0 00000008:00100000:0.0:1787827515.615165:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827515.615177:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564157696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827515.615209:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827515.615211:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827515.615212:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827515.615215:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2455 00000100:00080000:0.0:1787827515.615217:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827515.615219:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827515.615222:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827515.615233:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564157824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827515.615239:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827515.615241:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827515.615242:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827515.615244:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2455 00000100:00080000:0.0:1787827515.615246:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827515.615250:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827515.615253:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827515.615261:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564157952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827515.615321:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827515.615388:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827515/real 1787827515] req@ffff976882aef0c0 x1874670564157440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827515.615419:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564157440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827515.615438:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827515.615445:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827515.615452:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564157440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827515.615493:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564157568:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827515.615541:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827515.615562:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827515/real 1787827515] req@ffff976882aee940 x1874670564157568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827515.615583:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564157568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827515.615597:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827515.615602:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827515.615627:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564157568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827515.615639:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564157696:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827515.615671:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827515.615688:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827515/real 1787827515] req@ffff976882aee580 x1874670564157696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827515.615718:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564157696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827515.615739:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827515.615744:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827515.615749:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564157696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827515.615760:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564157824:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827515.615790:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827515.615806:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827515/real 1787827515] req@ffff976882aeed00 x1874670564157824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827515.615865:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564157824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827515.615879:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827515.615885:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827515.615890:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564157824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827515.615900:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564157952:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827515.615931:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827515.615948:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827515/real 1787827515] req@ffff976882aec3c0 x1874670564157952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827515.615967:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564157952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827515.615991:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827515.615996:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827515.616002:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564157952:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827515.870940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827515.870940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827515.870970:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827516.638914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827516.894889:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827516.894920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827516.894944:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827517.662948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827517.918962:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827517.918963:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827517.919000:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827518.686922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827518.942894:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827518.942894:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827518.942932:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827519.710957:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827519.966934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827519.966936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827519.966949:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827520.734964:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827520.734994:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827520.735004:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827520.735011:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827520.735027:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2460 00000100:00080000:1.0:1787827520.735043:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827520.735066:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827520.735117:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564158080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827520.735167:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827520.735171:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827520.735172:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827520.735176:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2460 00000100:00080000:1.0:1787827520.735180:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827520.735185:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827520.735196:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827520.735208:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564158208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827520.735221:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564158080:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827520.735224:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827520.735239:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827520.735242:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827520.735252:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2460 00000100:00080000:1.0:1787827520.735257:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827520.735262:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827520.735268:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827520.735286:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564158336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827520.735298:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827520.735302:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827520.735305:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827520.735310:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2460 00000100:00080000:1.0:1787827520.735315:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827520.735324:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787827520.735330:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827520.735336:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827520.735351:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564158464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827520.735360:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827520.735363:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827520.735364:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827520.735367:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2460 00000100:00080000:1.0:1787827520.735370:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827520.735374:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827520.735380:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827520.735385:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827520/real 1787827520] req@ffff976889efc780 x1874670564158080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827520.735395:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564158592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827520.735419:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564158080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827520.735430:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827520.735434:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827520.735437:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564158080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827520.735463:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564158208:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827520.735486:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827520.735496:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827520/real 1787827520] req@ffff976889eff480 x1874670564158208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827520.735511:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564158208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827520.735517:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827520.735520:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827520.735522:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564158208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827520.735527:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564158336:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827520.735538:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827520.735545:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827520/real 1787827520] req@ffff976889efe580 x1874670564158336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827520.735553:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564158336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827520.735559:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827520.735561:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827520.735563:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564158336:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827520.735567:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564158464:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827520.735600:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827520.735608:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827520/real 1787827520] req@ffff976889efe1c0 x1874670564158464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827520.735646:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564158464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827520.735659:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827520.735662:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827520.735665:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564158464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827520.735670:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564158592:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827520.735686:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827520.735694:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827520/real 1787827520] req@ffff976889efcb40 x1874670564158592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827520.735703:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564158592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827520.735710:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827520.735713:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827520.735715:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564158592:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827520.990952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827520.990952:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827520.991022:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827521.758961:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827522.014937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827522.014937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827522.014971:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827522.782933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827523.038900:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827523.038922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827523.038956:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827523.808081:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827524.062944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827524.062945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827524.062988:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827524.830933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827525.086928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827525.086928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827525.086994:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827525.854880:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827525.854893:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827525.854898:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827525.854902:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827525.854910:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2465 00000100:00080000:1.0:1787827525.854938:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827525.854944:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827525.854977:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564158720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827525.854999:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827525.855002:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827525.855003:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827525.855006:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2465 00000100:00080000:1.0:1787827525.855009:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827525.855011:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827525.855019:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827525.855029:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564158848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827525.855038:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827525.855039:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564158720:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827525.855042:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827525.855046:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827525.855049:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2465 00000100:00080000:1.0:1787827525.855051:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827525.855055:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827525.855064:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827525.855073:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564158976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827525.855080:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827525.855083:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827525.855084:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827525.855087:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2465 00000100:00080000:1.0:1787827525.855089:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827525.855091:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827525.855094:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827525.855103:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564159104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827525.855110:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787827525.855113:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827525.855116:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827525.855118:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827525.855122:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2465 00000100:00080000:1.0:1787827525.855125:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827525.855128:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827525.855130:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827525.855137:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564159232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827525.855168:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827525/real 1787827525] req@ffff976889efc3c0 x1874670564158720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827525.855189:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564158720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827525.855202:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827525.855207:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827525.855212:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564158720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827525.855238:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564158848:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827525.855262:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827525.855276:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827525/real 1787827525] req@ffff976889efe940 x1874670564158848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827525.855290:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564158848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827525.855299:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827525.855303:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827525.855307:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564158848:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827525.855313:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564158976:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827525.855335:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827525.855347:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827525/real 1787827525] req@ffff976889effc00 x1874670564158976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827525.855359:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564158976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827525.855369:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827525.855372:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827525.855376:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564158976:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827525.855383:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564159104:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827525.855405:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827525.855418:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827525/real 1787827525] req@ffff976889eff0c0 x1874670564159104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827525.855431:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564159104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827525.855440:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827525.855444:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827525.855447:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564159104:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827525.855454:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564159232:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827525.855473:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827525.855485:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827525/real 1787827525] req@ffff976889efed00 x1874670564159232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827525.855497:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564159232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827525.855506:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827525.855509:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827525.855513:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564159232:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827526.110899:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827526.110899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827526.110939:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827526.879012:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827527.134907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827527.134908:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827527.134915:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827527.902868:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827528.158872:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827528.158886:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827528.158948:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827528.926991:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827529.182957:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827529.182957:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827529.182983:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827529.950906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827530.206897:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827530.206906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827530.206919:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827530.974915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827530.974957:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827530.974962:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827530.975007:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827530.975015:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2470 00000100:00080000:0.0:1787827530.975023:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827530.975029:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827530.975060:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564159360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827530.975083:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827530.975085:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827530.975087:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827530.975089:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2470 00000100:00080000:0.0:1787827530.975091:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827530.975094:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827530.975102:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827530.975112:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564159488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827530.975120:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827530.975122:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827530.975126:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827530.975127:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564159360:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787827530.975130:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2470 00000100:00080000:0.0:1787827530.975136:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827530.975139:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827530.975142:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827530.975157:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564159616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827530.975169:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827530.975171:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827530.975173:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827530.975175:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2470 00000100:00080000:0.0:1787827530.975178:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827530.975181:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827530.975185:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1787827530.975189:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827530.975197:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564159744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827530.975204:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827530.975207:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827530.975208:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827530.975210:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2470 00000100:00080000:0.0:1787827530.975213:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827530.975215:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827530.975218:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827530.975226:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827530/real 1787827530] req@ffff976882aed680 x1874670564159360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827530.975231:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564159872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827530.975246:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564159360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827530.975255:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827530.975258:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827530.975261:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564159360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827530.975281:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564159488:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827530.975295:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827530.975304:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827530/real 1787827530] req@ffff976882aec3c0 x1874670564159488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827530.975312:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564159488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827530.975318:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827530.975320:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827530.975322:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564159488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827530.975326:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564159616:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827530.975335:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827530.975341:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827530/real 1787827530] req@ffff976882aeed00 x1874670564159616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827530.975349:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564159616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827530.975354:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827530.975356:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827530.975357:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564159616:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827530.975361:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564159744:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827530.975371:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827530.975376:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827530/real 1787827530] req@ffff976882aee580 x1874670564159744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827530.975383:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564159744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827530.975388:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827530.975390:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827530.975392:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564159744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827530.975395:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564159872:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827530.975403:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827530.975409:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827530/real 1787827530] req@ffff976882aee940 x1874670564159872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827530.975415:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564159872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827530.975435:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827530.975437:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827530.975439:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564159872:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827531.230922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827531.230922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827531.230953:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827531.998895:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827532.254907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827532.254923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827532.254932:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827533.022915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827533.278922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827533.278922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827533.278955:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827534.046889:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827534.302925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827534.302929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827534.302948:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827535.070915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827535.326938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827535.326939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827535.326974:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827536.094969:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827536.094981:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1787827536.094982:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827536.094987:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827536.095002:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2475 00000100:00080000:3.0:1787827536.095015:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827536.095024:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827536.095079:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564160000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827536.095141:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827536.095147:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827536.095150:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827536.095156:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2475 00000100:00080000:3.0:1787827536.095161:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827536.095167:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827536.095180:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827536.095202:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564160128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827536.095227:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564160000:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827536.095250:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827536.095256:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827536.095259:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827536.095267:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2475 00000100:00080000:3.0:1787827536.095273:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827536.095278:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827536.095287:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827536.095321:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564160256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827536.095339:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827536.095343:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827536.095344:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827536.095348:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2475 00000800:00000100:2.0:1787827536.095356:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827536.095357:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827536.095368:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827536.095376:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827536.095390:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564160384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827536.095402:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827536.095407:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827536.095409:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827536.095417:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2475 00000100:00100000:2.0:1787827536.095418:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827536/real 1787827536] req@ffff97688cb5de00 x1874670564160000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827536.095421:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827536.095433:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827536.095445:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827536.095457:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564160000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827536.095490:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564160512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827536.095542:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827536.095550:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827536.095555:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564160000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827536.095607:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564160128:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827536.095652:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827536.095683:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827536/real 1787827536] req@ffff97688cb5cb40 x1874670564160128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827536.095701:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564160128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827536.095713:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827536.095717:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827536.095721:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564160128:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827536.095729:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564160256:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827536.095754:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827536.095767:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827536/real 1787827536] req@ffff97688cb5c000 x1874670564160256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827536.095780:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564160256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827536.095789:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827536.095793:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827536.095796:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564160256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827536.095803:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564160384:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827536.095821:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827536.095878:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827536/real 1787827536] req@ffff97688cb5d680 x1874670564160384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827536.095902:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564160384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827536.095916:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827536.095922:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827536.095929:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564160384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827536.095957:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564160512:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827536.095986:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827536.095999:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827536/real 1787827536] req@ffff97688cb5f0c0 x1874670564160512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827536.096029:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564160512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827536.096039:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827536.096043:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827536.096046:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564160512:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827536.350962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827536.350973:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827536.351058:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827537.118911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827537.374948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827537.374956:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827537.374989:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827538.142905:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827538.398890:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827538.398893:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827538.398956:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827539.166906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827539.422927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827539.422955:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827539.422976:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827540.191028:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827540.446911:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827540.446943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827540.446968:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827541.214892:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827541.214913:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827541.214918:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827541.214930:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2481 00000100:00100000:2.0:1787827541.214937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827541.214942:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827541.214962:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827541.214998:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564160640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827541.215021:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827541.215023:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827541.215025:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827541.215028:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2481 00000100:00080000:3.0:1787827541.215032:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827541.215036:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827541.215044:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827541.215050:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564160640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827541.215056:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564160768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827541.215066:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827541.215069:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827541.215071:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827541.215074:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2481 00000100:00080000:3.0:1787827541.215077:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827541.215081:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827541.215085:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827541.215104:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564160896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827541.215111:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827541.215113:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827541.215114:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827541.215117:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2481 00000100:00080000:3.0:1787827541.215120:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827541.215123:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827541.215126:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1787827541.215133:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827541.215135:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564161024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827541.215144:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827541.215146:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827541.215148:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827541.215152:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2481 00000100:00080000:3.0:1787827541.215155:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827541.215157:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827541.215160:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827541.215170:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564161152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827541.215180:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827541/real 1787827541] req@ffff97688cb5f840 x1874670564160640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827541.215204:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564160640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827541.215216:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827541.215221:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827541.215224:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564160640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827541.215260:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564160768:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827541.215288:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827541.215304:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827541/real 1787827541] req@ffff97688cb5c3c0 x1874670564160768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827541.215319:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564160768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827541.215331:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827541.215334:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827541.215337:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564160768:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827541.215342:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564160896:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827541.215359:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827541.215367:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827541/real 1787827541] req@ffff97688cb5e1c0 x1874670564160896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827541.215375:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564160896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827541.215381:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827541.215384:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827541.215386:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564160896:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827541.215393:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564161024:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827541.215415:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827541.215446:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827541/real 1787827541] req@ffff97688cb5c780 x1874670564161024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827541.215464:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564161024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827541.215485:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827541.215488:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827541.215492:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564161024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827541.215499:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564161152:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827541.215522:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827541.215539:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827541/real 1787827541] req@ffff97688cb5cf00 x1874670564161152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827541.215552:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564161152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827541.215561:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827541.215564:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827541.215567:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564161152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827541.470927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827541.470928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827541.470957:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827542.238965:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827542.494906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827542.494914:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827542.494927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827543.262933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827543.518930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827543.518930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827543.518964:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827544.286926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827544.542917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827544.542917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827544.542947:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827545.310876:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827545.566878:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827545.566897:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827545.566899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00080000:3.0:1787827546.334966:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827546.334975:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1787827546.334976:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827546.334980:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827546.334991:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2486 00000100:00080000:3.0:1787827546.335002:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827546.335009:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827546.335067:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564161280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827546.335099:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827546.335104:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827546.335106:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827546.335112:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2486 00000100:00080000:3.0:1787827546.335117:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827546.335122:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827546.335134:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564161280:192.168.203.139@tcp:250:kworker.0 00000008:00100000:3.0:1787827546.335135:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827546.335160:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564161408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827546.335175:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827546.335180:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827546.335182:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827546.335187:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2486 00000100:00080000:3.0:1787827546.335192:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827546.335196:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827546.335202:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827546.335203:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827546.335224:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564161536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827546.335237:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827546.335241:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827546.335244:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827546.335250:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2486 00000100:00080000:3.0:1787827546.335254:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827546.335257:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827546/real 1787827546] req@ffff97688cb5d2c0 x1874670564161280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827546.335263:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827546.335292:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827546.335301:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564161280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827546.335308:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564161664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827546.335313:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827546.335322:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:3.0:1787827546.335324:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827546.335328:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564161280:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827546.335330:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827546.335334:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827546.335355:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2486 00000100:00080000:3.0:1787827546.335360:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827546.335364:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827546.335371:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827546.335372:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564161408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827546.335407:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564161792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827546.335420:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827546.335458:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827546/real 1787827546] req@ffff97688cb5cf00 x1874670564161408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827546.335468:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564161408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827546.335474:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827546.335476:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827546.335478:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564161408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827546.335482:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564161536:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827546.335496:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827546.335503:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827546/real 1787827546] req@ffff97688cb5c780 x1874670564161536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827546.335510:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564161536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827546.335516:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827546.335518:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827546.335520:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564161536:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827546.335523:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564161664:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827546.335533:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827546.335538:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827546/real 1787827546] req@ffff97688cb5e1c0 x1874670564161664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827546.335545:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564161664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827546.335550:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827546.335552:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827546.335553:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564161664:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827546.335565:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564161792:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827546.335576:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827546.335582:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827546/real 1787827546] req@ffff97688cb5c3c0 x1874670564161792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827546.335593:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564161792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827546.335598:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827546.335600:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827546.335602:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564161792:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827546.590915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827546.590922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827546.591070:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827547.358886:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827547.614968:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827547.614968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827547.615028:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827548.382960:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827548.638952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827548.638952:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827548.639061:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827549.406913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827549.662921:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827549.662922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827549.662963:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827550.430956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827550.686935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827550.686957:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827550.687011:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827551.454915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827551.455044:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827551.455062:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827551.455071:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827551.455087:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2491 00000100:00080000:0.0:1787827551.455105:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827551.455116:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827551.455190:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564161920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827551.455243:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827551.455248:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827551.455251:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827551.455258:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2491 00000100:00080000:0.0:1787827551.455274:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827551.455281:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827551.455297:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827551.455307:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564161920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827551.455320:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564162048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827551.455349:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827551.455354:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827551.455357:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827551.455386:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2491 00000100:00080000:0.0:1787827551.455397:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827551.455403:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827551.455412:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827551.455428:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827551.455437:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564162176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827551.455454:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827551.455460:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827551.455464:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827551.455475:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2491 00000100:00080000:0.0:1787827551.455483:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827551.455490:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827551/real 1787827551] req@ffff976882aef0c0 x1874670564161920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827551.455496:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827551.455519:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827551.455558:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564161920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827551.455562:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564162304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827551.455603:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827551.455621:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827551.455622:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:0.0:1787827551.455637:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787827551.455640:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564161920:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787827551.455643:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827551.455668:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2491 00000100:00080000:0.0:1787827551.455675:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827551.455683:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827551.455691:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827551.455704:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564162048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827551.455711:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564162432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827551.455766:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827551.455795:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827551/real 1787827551] req@ffff976882aee940 x1874670564162048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827551.455805:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564162048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827551.455813:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827551.455815:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827551.455818:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564162048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827551.455825:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564162176:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827551.455864:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827551.455873:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827551/real 1787827551] req@ffff976882aee580 x1874670564162176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827551.455881:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564162176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827551.455888:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827551.455890:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827551.455909:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564162176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827551.455914:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564162304:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827551.455932:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827551.455940:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827551/real 1787827551] req@ffff976882aeed00 x1874670564162304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827551.455950:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564162304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827551.455956:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827551.455959:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827551.455961:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564162304:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827551.455976:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564162432:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827551.456004:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827551.456011:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827551/real 1787827551] req@ffff976882aec3c0 x1874670564162432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827551.456018:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564162432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827551.456024:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827551.456026:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827551.456027:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564162432:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827551.710924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827551.710952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827551.710997:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827552.478958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827552.734935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827552.734935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827552.735036:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827553.502885:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827553.758899:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827553.758899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827553.759027:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827554.526942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827554.782920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827554.782935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827554.782979:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827555.550981:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827555.806894:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827555.806895:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827555.806956:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827556.574985:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827556.574999:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827556.575006:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827556.575009:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827556.575017:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2496 00000100:00080000:2.0:1787827556.575025:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827556.575031:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827556.575062:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564162560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827556.575084:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827556.575087:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827556.575088:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827556.575091:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2496 00000100:00080000:2.0:1787827556.575093:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827556.575096:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827556.575104:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827556.575118:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564162560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827556.575118:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564162688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827556.575133:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827556.575136:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827556.575137:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827556.575140:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2496 00000100:00080000:2.0:1787827556.575143:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827556.575145:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827556.575148:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827556.575157:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564162816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827556.575164:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827556.575167:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827556.575169:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827556.575172:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2496 00000100:00080000:2.0:1787827556.575176:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827556.575179:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787827556.575184:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827556.575192:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827556.575208:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564162944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827556.575215:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827556/real 1787827556] req@ffff976889cf9e00 x1874670564162560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827556.575225:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827556.575229:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827556.575232:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827556.575237:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2496 00000100:00080000:2.0:1787827556.575241:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827556.575246:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827556.575252:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827556.575255:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564162560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827556.575266:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787827556.575271:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564163072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827556.575272:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827556.575279:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564162560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827556.575311:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564162688:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827556.575358:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827556.575370:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827556/real 1787827556] req@ffff976889cfa940 x1874670564162688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827556.575379:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564162688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827556.575386:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827556.575388:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827556.575390:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564162688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827556.575395:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564162816:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827556.575408:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827556.575415:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827556/real 1787827556] req@ffff976889cfbc00 x1874670564162816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827556.575423:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564162816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827556.575428:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827556.575430:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827556.575432:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564162816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827556.575435:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564162944:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827556.575446:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827556.575452:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827556/real 1787827556] req@ffff976889cf8000 x1874670564162944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827556.575459:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564162944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827556.575464:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827556.575466:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827556.575468:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564162944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827556.575471:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564163072:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827556.575481:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827556.575486:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827556/real 1787827556] req@ffff976889cfb0c0 x1874670564163072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827556.575494:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564163072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827556.575499:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827556.575501:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827556.575503:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564163072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827556.830926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827556.830934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827556.830939:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827557.598882:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827557.854894:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827557.854928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827557.854965:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827558.622867:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827558.878889:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827558.878890:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827558.878893:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827559.646895:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827559.902884:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827559.902885:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827559.902927:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827560.671874:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827560.926892:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827560.926907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827560.926909:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827561.694882:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827561.694897:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827561.694904:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827561.694909:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827561.694919:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2501 00000100:00080000:2.0:1787827561.694957:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827561.694971:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827561.695013:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564163200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827561.695062:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827561.695067:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827561.695069:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827561.695074:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2501 00000100:00080000:2.0:1787827561.695079:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827561.695083:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827561.695093:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827561.695103:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564163200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827561.695126:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564163328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827561.695136:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827561.695138:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827561.695139:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827561.695142:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2501 00000100:00080000:2.0:1787827561.695145:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827561.695147:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827561.695152:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827561.695161:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564163456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827561.695169:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827561.695171:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827561.695190:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827561.695194:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2501 00000100:00080000:2.0:1787827561.695197:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827561.695201:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:1.0:1787827561.695208:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:2.0:1787827561.695208:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827561.695223:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564163584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827561.695232:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827561.695235:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827561.695237:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827561.695241:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2501 00000100:00080000:2.0:1787827561.695245:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827561.695249:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827561.695252:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827561.695263:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564163712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827561.695273:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827561/real 1787827561] req@ffff976889cf8f00 x1874670564163200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827561.695328:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564163200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827561.695336:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827561.695339:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827561.695342:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564163200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827561.695360:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564163328:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827561.695379:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827561.695389:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827561/real 1787827561] req@ffff976889cfb0c0 x1874670564163328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827561.695398:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564163328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827561.695403:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827561.695405:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827561.695408:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564163328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827561.695411:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564163456:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827561.695422:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827561.695428:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827561/real 1787827561] req@ffff976889cf8000 x1874670564163456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827561.695452:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564163456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827561.695459:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827561.695461:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827561.695464:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564163456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827561.695468:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564163584:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827561.695483:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827561.695491:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827561/real 1787827561] req@ffff976889cfbc00 x1874670564163584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827561.695501:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564163584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827561.695513:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827561.695516:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827561.695519:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564163584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827561.695523:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564163712:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827561.695535:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827561.695542:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827561/real 1787827561] req@ffff976889cfa940 x1874670564163712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827561.695551:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564163712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827561.695563:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827561.695567:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827561.695571:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564163712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827561.950888:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827561.950895:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827561.950937:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827562.718902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827562.974961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827562.974966:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827562.974968:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827563.743026:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827563.998905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827563.998954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827563.999006:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827564.766920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827565.022937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827565.022964:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827565.022973:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827565.791059:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827566.046898:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827566.046933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827566.046934:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827566.814915:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827566.814920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827566.814928:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827566.814933:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827566.814941:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2506 00000100:00080000:0.0:1787827566.814950:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827566.814957:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827566.814988:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564163840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827566.815012:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827566.815014:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827566.815016:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827566.815018:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2506 00000100:00080000:0.0:1787827566.815021:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827566.815023:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827566.815037:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827566.815049:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564163968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827566.815060:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827566.815063:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827566.815065:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827566.815068:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2506 00000100:00080000:0.0:1787827566.815070:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827566.815079:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564163840:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787827566.815081:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827566.815091:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827566.815100:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564164096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827566.815108:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827566.815110:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827566.815111:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827566.815115:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2506 00000100:00080000:0.0:1787827566.815118:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827566.815121:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827566.815124:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827566.815131:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564164224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827566.815137:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827566.815140:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827566.815142:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827566.815145:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2506 00000100:00080000:0.0:1787827566.815174:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827566.815176:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827566.815179:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827566.815195:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827566.815197:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564164352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827566.815293:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827566/real 1787827566] req@ffff976882aed680 x1874670564163840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827566.815314:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564163840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827566.815326:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827566.815330:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827566.815334:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564163840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827566.815362:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564163968:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827566.815410:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827566.815424:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827566/real 1787827566] req@ffff976882aec3c0 x1874670564163968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827566.815435:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564163968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827566.815443:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827566.815446:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827566.815450:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564163968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827566.815455:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564164096:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827566.815470:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827566.815478:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827566/real 1787827566] req@ffff976882aeed00 x1874670564164096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827566.815488:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564164096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827566.815495:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827566.815498:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827566.815500:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564164096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827566.815506:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564164224:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827566.815521:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827566.815529:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827566/real 1787827566] req@ffff976882aee580 x1874670564164224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827566.815539:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564164224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827566.815546:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827566.815549:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827566.815551:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564164224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827566.815556:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564164352:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827566.815569:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827566.815583:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827566/real 1787827566] req@ffff976882aee940 x1874670564164352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827566.815592:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564164352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827566.815599:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827566.815602:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827566.815604:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564164352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827567.070920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827567.070949:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827567.070990:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827567.838952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827568.094941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827568.094965:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827568.095140:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827568.862949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827569.118904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827569.118910:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827569.118998:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827569.886941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827570.142888:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827570.142892:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827570.142954:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827570.910911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827571.166980:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827571.166990:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827571.167079:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827571.934982:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827571.935017:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827571.935022:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827571.935036:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2511 00000100:00100000:2.0:1787827571.935042:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827571.935051:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827571.935064:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827571.935122:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564164480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827571.935154:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827571.935159:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827571.935161:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827571.935165:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2511 00000100:00080000:0.0:1787827571.935170:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827571.935175:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827571.935187:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827571.935206:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564164608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827571.935246:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827571.935251:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827571.935253:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827571.935260:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2511 00000100:00080000:0.0:1787827571.935276:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827571.935286:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827571.935286:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564164480:192.168.203.139@tcp:250:kworker.0 00000008:00100000:0.0:1787827571.935300:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827571.935320:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564164736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827571.935332:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827571.935338:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827571.935340:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827571.935346:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2511 00000100:00080000:0.0:1787827571.935349:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827571.935355:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827571.935361:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827571.935374:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564164864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827571.935392:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827571.935397:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827571.935399:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827571.935411:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2511 00000100:00080000:0.0:1787827571.935416:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827571.935423:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827571.935427:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827571.935434:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827571.935444:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564164992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827571.935510:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827571/real 1787827571] req@ffff976882aef0c0 x1874670564164480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827571.935550:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564164480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827571.935564:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827571.935569:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827571.935574:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564164480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827571.935625:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564164608:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827571.935653:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827571.935669:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827571/real 1787827571] req@ffff976882aee940 x1874670564164608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827571.935701:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564164608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827571.935730:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827571.935734:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827571.935738:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564164608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827571.935746:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564164736:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827571.935788:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827571.935800:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827571/real 1787827571] req@ffff976882aee580 x1874670564164736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827571.935825:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564164736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827571.935880:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827571.935885:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827571.935905:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564164736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827571.935912:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564164864:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827571.935932:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827571.935944:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827571/real 1787827571] req@ffff976882aeed00 x1874670564164864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827571.935957:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564164864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827571.935982:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827571.935987:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827571.935991:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564164864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827571.935998:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564164992:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827571.936027:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827571.936038:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827571/real 1787827571] req@ffff976882aec3c0 x1874670564164992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827571.936053:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564164992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827571.936080:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827571.936084:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827571.936089:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564164992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827572.190923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827572.190923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827572.191054:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827572.958908:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827573.214922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827573.214963:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827573.215029:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827573.982926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827574.238959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827574.238968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827574.239056:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827575.006934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827575.262935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827575.262956:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827575.263012:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827576.030915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827576.286887:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827576.286902:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827576.287028:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827577.054911:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827577.054916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827577.054919:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827577.054924:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827577.054932:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2516 00000100:00080000:0.0:1787827577.054941:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827577.054948:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827577.054984:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564165120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827577.055011:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827577.055015:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827577.055018:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827577.055023:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2516 00000100:00080000:0.0:1787827577.055028:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827577.055033:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827577.055043:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827577.055044:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564165120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827577.055063:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564165248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827577.055077:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827577.055081:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827577.055083:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827577.055086:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2516 00000100:00080000:0.0:1787827577.055097:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827577.055102:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827577.055107:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827577.055121:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564165376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827577.055132:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827577.055136:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787827577.055137:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564165248:192.168.203.139@tcp:38:kworker.0 00000100:00080000:0.0:1787827577.055140:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827577.055147:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2516 00000100:00080000:0.0:1787827577.055150:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827577.055154:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827577.055159:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827577.055168:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564165376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827577.055175:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564165504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827577.055200:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827577.055204:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827577.055206:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827577.055211:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2516 00000100:00080000:0.0:1787827577.055215:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827577.055217:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564165504:192.168.203.139@tcp:8:kworker.0 00000100:00080000:0.0:1787827577.055223:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827577.055231:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827577.055256:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564165632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827577.055304:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564165632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827577.310906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827577.310912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827577.310944:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827578.078987:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827578.334974:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827578.335005:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827578.335037:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827579.102987:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827579.358976:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827579.358999:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827579.359048:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827580.127023:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:1.0:1787827580.127117:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:1.0:1787827580.127175:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787827580.127193:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827580.127240:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827580.127252:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827580.127261:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827580.127271:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:0.0:1787827580.127338:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827577/real 1787827580] req@ffff976882aee940 x1874670564165632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827580.127381:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564165632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827580.127396:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827580.127402:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827580.127407:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564165632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827580.127417:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827577/real 1787827580] req@ffff976882aee580 x1874670564165504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827580.127433:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564165504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827580.127441:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827580.127445:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827580.127448:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564165504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827580.127456:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827577/real 1787827580] req@ffff976882aec3c0 x1874670564165248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827580.127468:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564165248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827580.127477:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827580.127481:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827580.127485:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564165248:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827580.127504:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827577/real 1787827580] req@ffff976882aeed00 x1874670564165376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827580.127518:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564165376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827580.127527:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827580.127531:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827580.127535:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564165376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827580.127542:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827577/real 1787827580] req@ffff976882aed680 x1874670564165120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827580.127564:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564165120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827580.127572:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827580.127576:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827580.127579:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564165120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827580.382971:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827580.382972:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827580.383068:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827581.151013:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827581.406937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827581.406940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827581.407088:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827582.174939:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827582.174944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827582.174955:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827582.174960:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827582.174972:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2522 00000100:00080000:0.0:1787827582.174986:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827582.174994:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827582.175035:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564165760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827582.175065:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827582.175069:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827582.175071:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827582.175076:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2522 00000100:00080000:0.0:1787827582.175081:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827582.175087:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827582.175103:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564165760:192.168.203.139@tcp:250:kworker.0 00000008:00100000:0.0:1787827582.175108:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827582.175131:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564165888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827582.175158:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827582.175162:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827582.175163:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827582.175167:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2522 00000100:00080000:0.0:1787827582.175180:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827582.175187:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:2.0:1787827582.175196:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:0.0:1787827582.175203:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827582.175225:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564166016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827582.175237:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827582.175241:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827582.175243:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827582.175250:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2522 00000100:00080000:0.0:1787827582.175259:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827582.175260:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827582/real 1787827582] req@ffff976882aed680 x1874670564165760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827582.175272:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827582.175287:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827582.175312:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564166144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827582.175314:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564165760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827582.175364:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827582.175368:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827582.175371:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827582.175378:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827582.175385:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:0.0:1787827582.175392:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2522 00000100:00080000:0.0:1787827582.175400:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827582.175407:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564165760:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787827582.175412:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827582.175426:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827582.175445:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564166272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827582.175461:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564165888:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827582.175501:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827582.175519:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827582/real 1787827582] req@ffff976882aeed00 x1874670564165888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827582.175544:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564165888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827582.175553:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827582.175557:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827582.175561:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564165888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827582.175569:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564166016:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827582.175591:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827582.175602:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827582/real 1787827582] req@ffff976882aec3c0 x1874670564166016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827582.175614:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564166016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827582.175622:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827582.175625:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827582.175628:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564166016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827582.175635:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564166144:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827582.175655:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827582.175667:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827582/real 1787827582] req@ffff976882aee580 x1874670564166144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827582.175679:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564166144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827582.175693:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827582.175697:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827582.175701:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564166144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827582.175721:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564166272:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827582.175743:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827582.175756:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827582/real 1787827582] req@ffff976882aee940 x1874670564166272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827582.175776:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564166272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827582.175786:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827582.175789:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827582.175793:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564166272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827582.430947:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827582.430965:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827582.431062:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827583.199049:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827583.454937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827583.454940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827583.454967:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827584.222917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827584.478905:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827584.478965:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827584.478988:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827585.247026:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827585.502933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827585.502933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827585.503028:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827586.270907:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827586.526906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827586.526968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827586.526977:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827587.294955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827587.294975:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827587.294988:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827587.294996:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827587.295012:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2527 00000100:00080000:2.0:1787827587.295030:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827587.295040:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827587.295134:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564166400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827587.295197:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827587.295214:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827587.295218:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827587.295226:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2527 00000100:00080000:2.0:1787827587.295232:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827587.295242:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827587.295259:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827587.295283:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564166528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827587.295289:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564166400:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787827587.295308:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827587.295315:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827587.295319:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827587.295326:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2527 00000100:00080000:2.0:1787827587.295332:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827587.295339:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827587.295359:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827587.295382:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564166656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827587.295396:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827587.295400:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827587.295402:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827587.295410:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2527 00000800:00000100:0.0:1787827587.295421:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787827587.295422:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827587.295432:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827587.295439:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827587.295455:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564166784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827587.295464:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827587.295467:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827587.295469:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827587.295473:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2527 00000100:00080000:2.0:1787827587.295478:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827587.295482:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827587.295497:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827587.295508:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564166912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827587.295514:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827587/real 1787827587] req@ffff976889cf9e00 x1874670564166400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827587.295585:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564166400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827587.295603:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827587.295610:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827587.295631:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564166400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827587.295658:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564166528:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827587.295687:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827587.295703:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827587/real 1787827587] req@ffff976889cfb480 x1874670564166528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827587.295724:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564166528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827587.295733:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827587.295737:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827587.295741:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564166528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827587.295748:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564166656:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827587.295771:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827587.295784:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827587/real 1787827587] req@ffff976889cfa580 x1874670564166656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827587.295798:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564166656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827587.295806:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827587.295810:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827587.295814:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564166656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827587.295821:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564166784:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827587.295870:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827587.295884:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827587/real 1787827587] req@ffff976889cfa1c0 x1874670564166784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827587.295899:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564166784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827587.295915:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827587.295919:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827587.295923:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564166784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827587.295930:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564166912:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827587.295953:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827587.295966:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827587/real 1787827587] req@ffff976889cf92c0 x1874670564166912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827587.295999:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564166912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827587.296013:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827587.296018:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827587.296023:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564166912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827587.551006:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827587.551047:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827587.551151:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827588.318938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827588.574898:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827588.574920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827588.574952:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827589.342934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827589.598985:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827589.598986:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827589.599053:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827590.366923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827590.622923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827590.622935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827590.622988:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827591.390977:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827591.646918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827591.646931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827591.646954:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827592.414929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827592.414972:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827592.414978:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827592.414982:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827592.414993:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2532 00000100:00080000:0.0:1787827592.415005:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827592.415013:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827592.415057:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564167040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827592.415100:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827592.415103:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827592.415104:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827592.415108:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2532 00000100:00080000:0.0:1787827592.415111:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827592.415115:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827592.415126:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827592.415132:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564167040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827592.415142:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564167168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827592.415150:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827592.415153:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827592.415154:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827592.415157:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2532 00000100:00080000:0.0:1787827592.415159:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827592.415162:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827592.415166:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827592.415174:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564167296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827592.415185:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827592.415188:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827592.415190:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827592.415192:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2532 00000100:00080000:0.0:1787827592.415195:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827592.415197:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827592.415200:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1787827592.415206:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827592.415209:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564167424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827592.415217:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827592.415220:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827592.415221:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827592.415224:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2532 00000100:00080000:0.0:1787827592.415227:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827592.415230:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827592.415233:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827592.415241:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564167552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827592.415261:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827592/real 1787827592] req@ffff976882aef0c0 x1874670564167040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827592.415282:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564167040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827592.415295:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827592.415301:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827592.415305:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564167040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827592.415332:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564167168:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827592.415353:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827592.415363:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827592/real 1787827592] req@ffff976882aee940 x1874670564167168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827592.415371:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564167168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827592.415378:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827592.415380:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827592.415383:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564167168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827592.415387:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564167296:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827592.415398:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827592.415404:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827592/real 1787827592] req@ffff976882aee580 x1874670564167296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827592.415411:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564167296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827592.415417:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827592.415419:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827592.415421:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564167296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827592.415425:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564167424:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827592.415437:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827592.415442:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827592/real 1787827592] req@ffff976882aec3c0 x1874670564167424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827592.415450:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564167424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827592.415455:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827592.415457:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827592.415458:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564167424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827592.415462:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564167552:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827592.415471:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827592.415477:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827592/real 1787827592] req@ffff976882aeed00 x1874670564167552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827592.415484:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564167552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827592.415489:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827592.415491:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827592.415492:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564167552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827592.670998:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827592.670999:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827592.671185:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827593.438935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827593.694983:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827593.694986:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827593.695106:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827594.462913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827594.718986:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827594.718986:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827594.719082:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827595.486954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827595.742978:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827595.742978:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827595.743007:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827596.510946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827596.766978:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827596.766978:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827596.767034:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827597.534975:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827597.534981:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827597.534996:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827597.535002:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827597.535015:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2537 00000100:00080000:0.0:1787827597.535027:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827597.535035:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827597.535125:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564167680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827597.535154:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827597.535158:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827597.535160:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827597.535163:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2537 00000100:00080000:0.0:1787827597.535166:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827597.535170:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827597.535179:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827597.535191:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564167808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827597.535201:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827597.535205:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827597.535207:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827597.535215:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2537 00000100:00100000:2.0:1787827597.535216:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564167680:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787827597.535220:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827597.535234:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827597.535240:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827597.535252:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564167936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827597.535260:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827597.535263:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827597.535265:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827597.535268:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2537 00000100:00080000:0.0:1787827597.535271:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827597.535274:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827597.535279:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827597.535298:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564168064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827597.535306:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827597.535309:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827597.535310:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827597.535314:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2537 00000100:00080000:0.0:1787827597.535317:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827597.535320:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827597.535326:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827597.535335:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564168192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827597.535358:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827597.535436:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827597/real 1787827597] req@ffff976882aed680 x1874670564167680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827597.535468:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564167680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827597.535488:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827597.535496:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827597.535503:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564167680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827597.535542:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564167808:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827597.535582:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827597.535603:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827597/real 1787827597] req@ffff976882aeed00 x1874670564167808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827597.535625:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564167808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827597.535655:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827597.535660:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827597.535665:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564167808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827597.535674:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564167936:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827597.535699:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827597.535712:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827597/real 1787827597] req@ffff976882aec3c0 x1874670564167936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827597.535728:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564167936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827597.535740:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827597.535745:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827597.535750:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564167936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827597.535758:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564168064:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827597.535782:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827597.535796:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827597/real 1787827597] req@ffff976882aee580 x1874670564168064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827597.535812:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564168064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827597.535823:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827597.535828:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827597.535833:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564168064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827597.535841:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564168192:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827597.535905:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827597.535919:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827597/real 1787827597] req@ffff976882aee940 x1874670564168192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827597.535969:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564168192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827597.535984:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827597.535990:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827597.535996:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564168192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827597.790973:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827597.790979:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827597.791070:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827598.558933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827598.815006:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827598.815013:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827598.815045:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827599.582926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827599.838950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827599.838988:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827599.839044:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827600.607126:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827600.863041:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827600.863063:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827600.863121:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827601.630901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827601.886961:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827601.886961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827601.887056:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827602.654940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827602.654946:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827602.654954:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827602.654958:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827602.654968:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2542 00000100:00080000:1.0:1787827602.654980:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827602.654987:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827602.655049:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564168320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827602.655074:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827602.655077:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827602.655078:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827602.655082:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2542 00000100:00080000:1.0:1787827602.655086:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827602.655090:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827602.655101:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827602.655106:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564168320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827602.655116:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564168448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827602.655125:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827602.655128:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827602.655129:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827602.655132:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2542 00000100:00080000:1.0:1787827602.655135:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827602.655137:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827602.655146:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827602.655155:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564168576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827602.655161:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827602.655163:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827602.655165:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827602.655168:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2542 00000100:00080000:1.0:1787827602.655178:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827602.655181:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827602.655185:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1787827602.655194:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827602.655195:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564168704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827602.655204:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827602.655207:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827602.655209:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827602.655212:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2542 00000100:00080000:1.0:1787827602.655216:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827602.655221:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827602.655226:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827602.655240:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564168832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827602.655255:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827602/real 1787827602] req@ffff976889efed00 x1874670564168320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827602.655271:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564168320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827602.655281:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827602.655284:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827602.655288:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564168320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827602.655314:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564168448:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827602.655340:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827602.655355:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827602/real 1787827602] req@ffff976889eff0c0 x1874670564168448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827602.655370:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564168448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827602.655377:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827602.655380:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827602.655382:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564168448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827602.655387:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564168576:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827602.655403:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827602.655411:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827602/real 1787827602] req@ffff976889effc00 x1874670564168576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827602.655418:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564168576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827602.655424:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827602.655427:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827602.655429:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564168576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827602.655432:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564168704:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827602.655444:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827602.655450:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827602/real 1787827602] req@ffff976889efe940 x1874670564168704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827602.655458:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564168704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827602.655463:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827602.655465:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827602.655467:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564168704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827602.655470:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564168832:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827602.655480:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827602.655485:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827602/real 1787827602] req@ffff976889efc3c0 x1874670564168832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827602.655491:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564168832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827602.655496:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827602.655498:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827602.655500:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564168832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827602.910947:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827602.910948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827602.911056:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827603.678937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827603.934925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827603.934949:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827603.935103:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827604.703000:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827604.958940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787827604.958940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827604.959081:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827605.726937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827605.982921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827605.982924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827605.983006:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827606.750962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827607.006915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827607.006941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827607.007013:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827607.774918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827607.774981:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827607.774990:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827607.774994:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827607.775003:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2547 00000100:00080000:3.0:1787827607.775012:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827607.775018:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827607.775054:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564168960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827607.775085:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827607.775090:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827607.775092:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827607.775097:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2547 00000100:00080000:3.0:1787827607.775103:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827607.775108:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827607.775118:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827607.775124:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564168960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827607.775148:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564169088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827607.775160:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827607.775163:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827607.775165:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827607.775168:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2547 00000100:00080000:3.0:1787827607.775171:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827607.775174:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827607.775178:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827607.775186:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564169216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827607.775192:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827607.775194:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827607.775195:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827607.775198:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2547 00000100:00080000:3.0:1787827607.775201:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827607.775203:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827607.775207:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827607.775213:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564169344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827607.775227:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827607.775227:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827607.775236:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827607.775239:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827607.775242:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2547 00000100:00080000:3.0:1787827607.775245:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827607.775247:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827607.775251:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827607.775262:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564169472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827607.775296:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827607/real 1787827607] req@ffff97688cb5f840 x1874670564168960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827607.775320:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564168960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827607.775332:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827607.775338:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827607.775343:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564168960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827607.775378:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564169088:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827607.775407:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827607.775421:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827607/real 1787827607] req@ffff97688cb5f0c0 x1874670564169088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827607.775430:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564169088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827607.775437:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827607.775440:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827607.775442:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564169088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827607.775447:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564169216:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827607.775461:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827607.775468:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827607/real 1787827607] req@ffff97688cb5d680 x1874670564169216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827607.775480:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564169216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827607.775486:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827607.775488:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827607.775489:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564169216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827607.775494:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564169344:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827607.775505:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827607.775511:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827607/real 1787827607] req@ffff97688cb5c000 x1874670564169344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827607.775518:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564169344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827607.775523:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827607.775525:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827607.775527:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564169344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827607.775530:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564169472:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827607.775540:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827607.775546:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827607/real 1787827607] req@ffff97688cb5cb40 x1874670564169472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827607.775552:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564169472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827607.775562:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827607.775564:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827607.775566:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564169472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827608.030948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827608.030955:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827608.031051:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827608.799010:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827609.055011:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827609.055046:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827609.055129:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827609.822920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827610.079048:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827610.079057:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827610.079080:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827610.847019:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827611.103054:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827611.103059:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827611.103138:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827611.870909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827612.126924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827612.126972:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827612.127050:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827612.895054:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827612.895116:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827612.895139:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827612.895147:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827612.895165:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2552 00000100:00080000:1.0:1787827612.895182:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827612.895204:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827612.895266:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564169600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827612.895336:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827612.895342:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827612.895347:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827612.895353:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2552 00000100:00080000:1.0:1787827612.895360:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827612.895367:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827612.895396:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827612.895412:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564169600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827612.895420:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564169728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827612.895439:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827612.895456:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827612.895461:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827612.895471:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2552 00000100:00080000:1.0:1787827612.895480:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827612.895491:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827612.895503:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827612.895531:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564169856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827612.895565:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787827612.895566:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827612.895578:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827612.895585:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827612.895603:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2552 00000100:00080000:1.0:1787827612.895620:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827612.895626:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827612.895637:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827612.895683:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564169984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827612.895685:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827612/real 1787827612] req@ffff976889efcb40 x1874670564169600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827612.895725:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827612.895739:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827612.895744:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827612.895750:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2552 00000100:00080000:1.0:1787827612.895756:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827612.895762:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827612.895772:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564169600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787827612.895773:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827612.895803:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564170112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827612.895808:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827612.895891:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827612.895903:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564169600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827612.896017:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564169728:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827612.896057:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827612.896096:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827612/real 1787827612] req@ffff976889efe1c0 x1874670564169728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827612.896125:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564169728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827612.896144:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827612.896152:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827612.896161:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564169728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827612.896179:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564169856:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827612.896219:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827612.896249:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827612/real 1787827612] req@ffff976889efe580 x1874670564169856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827612.896277:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564169856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827612.896291:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827612.896296:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827612.896319:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564169856:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827612.896328:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564169984:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827612.896359:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827612.896373:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827612/real 1787827612] req@ffff976889eff480 x1874670564169984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827612.896387:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564169984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827612.896409:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827612.896413:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827612.896417:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564169984:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827612.896424:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564170112:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827612.896443:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827612.896453:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827612/real 1787827612] req@ffff976889efc780 x1874670564170112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827612.896476:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564170112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827612.896487:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827612.896490:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827612.896494:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564170112:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827613.150920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827613.151010:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827613.151063:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827613.919001:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827614.175018:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827614.175018:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827614.175032:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827614.942909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827615.198967:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827615.198969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827615.199000:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827615.966978:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000008:00020000:3.0:1787827615.966980:0:42221:0:(osc_cache.c:999:osc_extent_wait()) extent ffff9768ad11eb40@{[0 -> 0/1023], [3|0|+|rpc|wiuY|ffff976886033a40], [28672|1|+|-|ffff976883514000|1024|ffff9768812f4f80]} lustre-OST0000-osc-ffff9768877a4800: wait ext to 0 timedout, recovery in progress? 00000008:00020000:3.0:1787827615.983206:0:42221:0:(osc_cache.c:999:osc_extent_wait()) ### extent: 000000005e3980cb ns: lustre-OST0000-osc-ffff9768877a4800 lock: 00000000c398e6de/0x8e7c99ee62ab1943 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800029480000000 nid: local remote: 0x22153cd396b83ce1 expref: -99 pid: 56198 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827616.222987:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827616.223021:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827616.223078:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827616.991003:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827617.246978:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827617.246978:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827617.247013:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827618.014998:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827618.015043:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827618.015054:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827618.015060:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827618.015073:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2557 00000100:00080000:3.0:1787827618.015085:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827618.015094:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827618.015172:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564170240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827618.015222:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827618.015225:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827618.015227:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827618.015231:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2557 00000100:00080000:3.0:1787827618.015234:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827618.015237:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827618.015248:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827618.015284:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564170368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827618.015296:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827618.015301:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827618.015310:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827618.015311:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564170240:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827618.015327:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2557 00000100:00080000:3.0:1787827618.015332:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827618.015336:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827618.015342:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827618.015355:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564170496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827618.015368:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827618.015371:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827618.015373:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827618.015377:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2557 00000100:00080000:3.0:1787827618.015380:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827618.015384:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827618.015388:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827618.015399:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564170624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827618.015406:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827618.015410:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827618.015412:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827618.015417:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2557 00000100:00080000:3.0:1787827618.015420:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827618.015423:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827618.015426:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827618.015437:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827618.015437:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564170752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827618.015529:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827618/real 1787827618] req@ffff97688cb5de00 x1874670564170240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827618.015551:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564170240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827618.015564:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827618.015569:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827618.015573:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564170240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827618.015609:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564170368:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827618.015637:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827618.015651:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827618/real 1787827618] req@ffff97688cb5cb40 x1874670564170368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827618.015664:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564170368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827618.015672:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827618.015676:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827618.015679:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564170368:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827618.015685:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564170496:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827618.015701:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827618.015710:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827618/real 1787827618] req@ffff97688cb5c000 x1874670564170496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827618.015721:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564170496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827618.015729:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827618.015732:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827618.015735:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564170496:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827618.015741:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564170624:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827618.015776:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827618.015789:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827618/real 1787827618] req@ffff97688cb5d680 x1874670564170624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827618.015804:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564170624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827618.015814:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827618.015817:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827618.015821:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564170624:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827618.015827:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564170752:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827618.015932:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827618.015949:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827618/real 1787827618] req@ffff97688cb5f0c0 x1874670564170752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827618.015961:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564170752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827618.015970:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827618.015973:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827618.015977:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564170752:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827618.270938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827618.271019:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827618.271028:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827619.038975:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827619.294974:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827619.295014:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827619.295075:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827620.063021:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827620.318957:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827620.318957:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827620.319024:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827621.086955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827621.342931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827621.342931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827621.343008:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827622.110921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827622.366928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827622.366987:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827622.366989:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827623.134910:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827623.134924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827623.134924:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827623.134930:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827623.134942:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2563 00000100:00080000:3.0:1787827623.134954:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827623.134979:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827623.135029:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564170880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827623.135056:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827623.135059:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827623.135061:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827623.135065:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2563 00000100:00080000:3.0:1787827623.135068:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827623.135072:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827623.135081:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827623.135110:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564171008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827623.135122:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564170880:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827623.135122:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827623.135133:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827623.135135:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827623.135138:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2563 00000100:00080000:3.0:1787827623.135141:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827623.135143:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827623.135148:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827623.135158:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564171136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827623.135165:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827623.135168:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827623.135170:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827623.135173:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2563 00000100:00080000:3.0:1787827623.135176:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827623.135179:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827623.135183:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1787827623.135189:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827623.135194:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564171264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827623.135205:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827623.135209:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827623.135212:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827623.135217:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2563 00000100:00080000:3.0:1787827623.135238:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827623.135243:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827623.135250:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827623.135253:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827623/real 1787827623] req@ffff97688cb5f840 x1874670564170880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827623.135273:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564171392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827623.135277:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564170880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827623.135315:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827623.135322:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827623.135329:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564170880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827623.135376:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564171008:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827623.135407:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827623.135420:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827623/real 1787827623] req@ffff97688cb5c3c0 x1874670564171008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827623.135431:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564171008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827623.135455:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827623.135458:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827623.135461:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564171008:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827623.135466:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564171136:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827623.135482:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827623.135491:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827623/real 1787827623] req@ffff97688cb5e1c0 x1874670564171136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827623.135517:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564171136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827623.135524:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827623.135526:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827623.135528:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564171136:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827623.135532:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564171264:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827623.135546:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827623.135556:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827623/real 1787827623] req@ffff97688cb5c780 x1874670564171264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827623.135570:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564171264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827623.135580:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827623.135584:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827623.135588:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564171264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827623.135596:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564171392:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827623.135619:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827623.135630:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827623/real 1787827623] req@ffff97688cb5cf00 x1874670564171392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827623.135639:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564171392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827623.135646:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827623.135648:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827623.135651:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564171392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827623.390946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827623.390948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827623.391021:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827624.158927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827624.415000:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827624.415002:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827624.415046:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827625.182929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827625.438939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827625.438960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827625.439070:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827626.206933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827626.462948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827626.462964:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827626.463045:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827627.230947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827627.486957:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827627.486960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827627.486981:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827628.254894:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827628.254900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827628.254903:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827628.254909:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827628.254917:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2568 00000100:00080000:3.0:1787827628.254926:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827628.254932:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827628.254960:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564171520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827628.254992:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827628.254994:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827628.254996:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827628.254999:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2568 00000100:00080000:3.0:1787827628.255001:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827628.255004:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827628.255011:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827628.255021:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564171648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827628.255030:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827628.255033:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827628.255034:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827628.255038:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2568 00000100:00100000:2.0:1787827628.255040:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564171520:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827628.255041:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827628.255046:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827628.255051:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827628.255059:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564171776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827628.255065:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827628.255067:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827628.255069:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827628.255071:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2568 00000100:00080000:3.0:1787827628.255074:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827628.255077:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827628.255081:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827628.255087:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564171904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827628.255093:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827628.255096:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827628.255097:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787827628.255102:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827628.255103:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2568 00000100:00080000:3.0:1787827628.255108:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827628.255111:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827628.255114:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827628.255122:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564172032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827628.255152:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827628/real 1787827628] req@ffff97688cb5d2c0 x1874670564171520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827628.255174:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564171520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827628.255183:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827628.255186:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827628.255189:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564171520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827628.255208:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564171648:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827628.255238:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827628.255252:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827628/real 1787827628] req@ffff97688cb5cf00 x1874670564171648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827628.255266:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564171648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827628.255276:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827628.255280:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827628.255283:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564171648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827628.255287:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564171776:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827628.255304:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827628.255312:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827628/real 1787827628] req@ffff97688cb5c780 x1874670564171776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827628.255319:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564171776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827628.255325:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827628.255327:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827628.255329:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564171776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827628.255333:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564171904:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827628.255344:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827628.255352:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827628/real 1787827628] req@ffff97688cb5e1c0 x1874670564171904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827628.255364:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564171904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827628.255373:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827628.255376:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827628.255380:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564171904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827628.255387:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564172032:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827628.255408:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827628.255416:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827628/real 1787827628] req@ffff97688cb5c3c0 x1874670564172032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827628.255424:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564172032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827628.255430:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827628.255432:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827628.255434:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564172032:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827628.510908:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827628.510925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827628.511000:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827629.278918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827629.534909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827629.534913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827629.534915:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827630.302981:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827630.558943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827630.558950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827630.559050:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827631.326973:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827631.582936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827631.583024:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827631.583026:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827632.350954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827632.606986:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827632.606986:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827632.607048:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827633.374995:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827633.375094:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827633.375104:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827633.375109:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827633.375121:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2573 00000100:00080000:1.0:1787827633.375136:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827633.375148:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827633.375206:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564172160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827633.375279:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827633.375282:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827633.375284:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827633.375288:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2573 00000100:00080000:1.0:1787827633.375291:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827633.375295:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827633.375305:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827633.375318:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564172288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827633.375329:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827633.375334:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827633.375336:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827633.375336:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564172160:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827633.375347:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2573 00000100:00080000:1.0:1787827633.375350:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827633.375353:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827633.375356:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827633.375376:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564172416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827633.375385:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827633.375388:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827633.375389:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827633.375393:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2573 00000100:00080000:1.0:1787827633.375398:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827633.375400:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827633.375405:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1787827633.375409:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827633.375420:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564172544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827633.375429:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827633.375432:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827633.375433:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827633.375437:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2573 00000100:00080000:1.0:1787827633.375441:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827633.375446:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827633.375450:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827633/real 1787827633] req@ffff976889efc780 x1874670564172160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787827633.375454:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827633.375476:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564172672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827633.375494:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564172160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827633.375511:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827633.375517:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827633.375525:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564172160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827633.375550:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564172288:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827633.375614:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827633.375632:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827633/real 1787827633] req@ffff976889eff480 x1874670564172288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827633.375664:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564172288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827633.375676:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827633.375680:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827633.375685:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564172288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827633.375693:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564172416:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827633.375717:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827633.375730:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827633/real 1787827633] req@ffff976889efe580 x1874670564172416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827633.375744:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564172416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827633.375766:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827633.375771:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827633.375775:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564172416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827633.375782:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564172544:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827633.375887:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827633.375905:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827633/real 1787827633] req@ffff976889efe1c0 x1874670564172544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827633.375923:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564172544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827633.375941:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827633.375951:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827633.375973:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564172544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827633.375982:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564172672:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827633.376008:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827633.376022:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827633/real 1787827633] req@ffff976889efcb40 x1874670564172672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827633.376033:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564172672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827633.376039:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827633.376042:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827633.376045:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564172672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827633.630958:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827633.630969:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827633.631068:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827634.399006:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827634.655022:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827634.655050:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827634.655052:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827635.422910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827635.678894:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827635.678902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827635.678941:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827636.446916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827636.702935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827636.702964:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827636.702987:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827637.470973:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827637.726904:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827637.726911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827637.726983:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827638.494914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827638.494965:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827638.494973:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827638.494978:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827638.494989:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2578 00000100:00080000:1.0:1787827638.495001:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827638.495009:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827638.495053:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564172800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827638.495089:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827638.495093:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827638.495095:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827638.495100:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2578 00000100:00080000:1.0:1787827638.495104:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827638.495108:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827638.495118:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827638.495131:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564172928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827638.495145:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827638.495150:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827638.495152:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827638.495156:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2578 00000100:00080000:1.0:1787827638.495160:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827638.495163:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827638.495169:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827638.495179:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564172800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827638.495194:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564173056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827638.495210:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827638.495214:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827638.495216:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827638.495221:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2578 00000100:00080000:1.0:1787827638.495225:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827638.495229:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827638.495235:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827638.495250:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564173184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827638.495260:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787827638.495269:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787827638.495270:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827638.495277:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827638.495283:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2578 00000100:00080000:1.0:1787827638.495288:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827638.495292:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827638.495298:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827638.495312:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564173312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827638.495328:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827638/real 1787827638] req@ffff976889efc3c0 x1874670564172800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827638.495351:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564172800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827638.495365:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827638.495370:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827638.495375:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564172800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827638.495385:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564172928:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827638.495421:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827638.495443:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827638/real 1787827638] req@ffff976889efe940 x1874670564172928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827638.495465:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564172928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827638.495480:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827638.495486:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827638.495491:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564172928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827638.495519:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564173056:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827638.495545:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827638.495557:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827638/real 1787827638] req@ffff976889effc00 x1874670564173056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827638.495578:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564173056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827638.495587:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827638.495591:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827638.495594:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564173056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827638.495599:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564173184:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827638.495617:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827638.495627:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827638/real 1787827638] req@ffff976889eff0c0 x1874670564173184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827638.495639:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564173184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827638.495647:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827638.495650:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827638.495653:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564173184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827638.495659:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564173312:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827638.495674:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827638.495683:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827638/real 1787827638] req@ffff976889efed00 x1874670564173312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827638.495694:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564173312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827638.495710:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827638.495713:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827638.495717:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564173312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827638.750913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827638.750925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827638.750991:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827639.518916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827639.774929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827639.774929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827639.775054:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827640.542901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827640.798921:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827640.798936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827640.798992:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827641.566920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827641.823011:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827641.823027:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827641.823027:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827642.590963:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827642.846919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827642.846936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827642.847001:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827643.614932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827643.614938:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827643.614954:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827643.614958:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827643.614983:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2583 00000100:00080000:1.0:1787827643.614992:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827643.614998:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827643.615043:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564173440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827643.615067:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827643.615070:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827643.615072:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827643.615075:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2583 00000100:00080000:1.0:1787827643.615077:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827643.615080:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827643.615088:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827643.615097:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564173568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827643.615104:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827643.615107:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827643.615108:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827643.615111:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2583 00000100:00080000:1.0:1787827643.615113:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827643.615118:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827643.615119:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564173440:192.168.203.139@tcp:250:kworker.0 00000008:00100000:1.0:1787827643.615122:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827643.615149:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564173696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827643.615166:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827643.615169:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827643.615170:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827643.615172:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2583 00000100:00080000:1.0:1787827643.615174:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827643.615176:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827643.615180:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827643.615187:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564173824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827643.615193:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827643.615194:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827643.615196:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827643.615198:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2583 00000100:00080000:1.0:1787827643.615202:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827643.615205:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827643.615207:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827643.615214:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564173952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827643.615257:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564173568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827643.615292:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564173696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827643.615308:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564173824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827643.615324:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564173952:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827643.870923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827643.870924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827643.870988:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827644.639015:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827644.894941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827644.894947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827644.894965:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827645.662996:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827645.918931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827645.918938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827645.919001:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827646.687137:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:1.0:1787827646.687307:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:1.0:1787827646.687364:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787827646.687382:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827646.687432:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827646.687442:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827646.687449:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827646.687457:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:2.0:1787827646.687497:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827643/real 1787827646] req@ffff976889effc00 x1874670564173696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827646.687525:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564173696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827646.687538:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827646.687543:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827646.687549:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564173696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827646.687557:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827643/real 1787827646] req@ffff976889efe940 x1874670564173824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827646.687571:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564173824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827646.687584:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827646.687587:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827646.687591:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564173824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827646.687598:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827643/real 1787827646] req@ffff976889efc3c0 x1874670564173952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827646.687609:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564173952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827646.687617:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827646.687620:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827646.687623:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564173952:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827646.687629:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827643/real 1787827646] req@ffff976889efed00 x1874670564173440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827646.687642:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564173440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827646.687650:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827646.687653:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827646.687657:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564173440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827646.687664:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827643/real 1787827646] req@ffff976889eff0c0 x1874670564173568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827646.687675:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564173568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827646.687682:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827646.687686:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827646.687689:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564173568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827646.942922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827646.942922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827646.942994:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827647.711043:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827647.966989:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827647.966993:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827647.967079:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827648.734926:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827648.734934:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827648.734938:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827648.734946:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2588 00000100:00080000:1.0:1787827648.734956:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827648.734963:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827648.734978:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827648.735000:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564174080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827648.735024:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827648.735029:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827648.735031:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827648.735037:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2588 00000100:00080000:1.0:1787827648.735042:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827648.735047:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827648.735052:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564174080:192.168.203.139@tcp:250:kworker.0 00000008:00100000:1.0:1787827648.735060:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827648.735079:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564174208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827648.735099:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827648.735103:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827648.735106:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827648.735112:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2588 00000800:00000100:0.0:1787827648.735118:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787827648.735119:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827648.735127:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827648.735133:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827648.735146:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564174336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827648.735156:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827648.735159:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827648.735162:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827648.735168:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2588 00000100:00080000:1.0:1787827648.735172:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827648.735175:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827648/real 1787827648] req@ffff976889efcb40 x1874670564174080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827648.735179:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827648.735187:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827648.735202:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564174464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827648.735208:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564174080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827648.735214:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827648.735222:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827648.735226:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827648.735233:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2588 00000100:00080000:0.0:1787827648.735234:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827648.735242:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827648.735243:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:1.0:1787827648.735253:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827648.735254:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564174080:192.168.203.139@tcp:250:kworker.0 00000008:00100000:1.0:1787827648.735275:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827648.735289:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564174208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827648.735299:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564174592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827648.735333:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827648.735360:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827648/real 1787827648] req@ffff976889efe1c0 x1874670564174208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827648.735373:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564174208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827648.735384:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827648.735388:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827648.735392:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564174208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827648.735401:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564174336:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827648.735423:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827648.735436:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827648/real 1787827648] req@ffff976889efe580 x1874670564174336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827648.735448:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564174336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827648.735457:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827648.735461:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827648.735464:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564174336:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827648.735469:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564174464:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827648.735491:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827648.735503:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827648/real 1787827648] req@ffff976889eff480 x1874670564174464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827648.735525:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564174464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827648.735534:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827648.735538:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827648.735541:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564174464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827648.735559:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564174592:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827648.735580:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827648.735592:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827648/real 1787827648] req@ffff976889efc780 x1874670564174592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827648.735602:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564174592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827648.735608:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827648.735610:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827648.735613:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564174592:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827648.990911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827648.990911:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827648.990986:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827649.759008:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827650.014917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827650.014917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827650.014955:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827650.782933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827651.038925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827651.038942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827651.038973:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827651.806965:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827652.062922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827652.062922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827652.062990:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827652.830941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827653.086906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827653.086906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827653.086948:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827653.854903:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827653.854907:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827653.854913:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827653.854918:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827653.854925:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2593 00000100:00080000:3.0:1787827653.854934:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827653.854940:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827653.854972:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564174720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827653.854992:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827653.855002:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827653.855003:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827653.855006:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2593 00000100:00080000:3.0:1787827653.855009:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827653.855011:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827653.855019:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827653.855027:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564174848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827653.855034:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827653.855036:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827653.855038:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827653.855040:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2593 00000100:00080000:3.0:1787827653.855041:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827653.855043:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827653.855046:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827653.855051:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564174976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827653.855058:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827653.855060:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827653.855061:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827653.855064:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2593 00000100:00080000:3.0:1787827653.855067:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827653.855069:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827653.855074:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827653.855078:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564174720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827653.855083:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564175104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827653.855096:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827653.855100:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827653.855101:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827653.855105:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2593 00000100:00080000:3.0:1787827653.855109:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827653.855113:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827653.855119:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827653.855132:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564175232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827653.855143:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827653.855183:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827653/real 1787827653] req@ffff97688cb5f840 x1874670564174720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827653.855201:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564174720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827653.855211:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827653.855215:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827653.855220:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564174720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827653.855227:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564174848:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827653.855248:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827653.855270:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827653/real 1787827653] req@ffff97688cb5f0c0 x1874670564174848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827653.855283:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564174848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827653.855294:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827653.855297:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827653.855301:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564174848:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827653.855307:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564174976:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827653.855325:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827653.855335:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827653/real 1787827653] req@ffff97688cb5d680 x1874670564174976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827653.855349:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564174976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827653.855358:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827653.855361:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827653.855365:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564174976:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827653.855390:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564175104:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827653.855408:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827653.855418:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827653/real 1787827653] req@ffff97688cb5c000 x1874670564175104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827653.855430:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564175104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827653.855439:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827653.855442:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827653.855446:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564175104:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827653.855452:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564175232:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827653.855467:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827653.855476:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827653/real 1787827653] req@ffff97688cb5cb40 x1874670564175232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827653.855487:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564175232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827653.855503:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827653.855507:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827653.855510:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564175232:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827654.110930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827654.110949:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827654.111021:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827654.878925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827655.134929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827655.134930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827655.134956:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827655.902961:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827656.158919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827656.158919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827656.158968:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827656.926922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827657.182889:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827657.182906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827657.182968:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827657.950882:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827658.206887:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827658.206891:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827658.206938:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827658.974930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827658.974933:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827658.974940:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827658.974944:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827658.974952:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2598 00000100:00080000:1.0:1787827658.974962:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827658.974968:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827658.975000:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564175360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827658.975029:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827658.975032:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827658.975033:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827658.975037:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2598 00000100:00080000:1.0:1787827658.975039:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827658.975042:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827658.975051:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827658.975060:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564175488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827658.975068:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827658.975070:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827658.975071:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827658.975074:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2598 00000100:00080000:1.0:1787827658.975076:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827658.975080:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827658.975082:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564175360:192.168.203.139@tcp:250:kworker.0 00000008:00100000:1.0:1787827658.975083:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827658.975093:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564175616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827658.975101:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827658.975103:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827658.975105:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827658.975108:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2598 00000100:00080000:1.0:1787827658.975109:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827658.975112:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827658.975115:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827658.975123:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564175744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827658.975130:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827658.975132:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827658.975133:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827658.975136:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2598 00000100:00080000:1.0:1787827658.975138:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827658.975140:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827658.975143:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787827658.975148:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827658.975152:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564175872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827658.975202:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827658/real 1787827658] req@ffff976889efc780 x1874670564175360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827658.975220:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564175360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827658.975231:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827658.975236:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827658.975239:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564175360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827658.975247:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564175488:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827658.975272:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827658.975287:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827658/real 1787827658] req@ffff976889eff480 x1874670564175488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827658.975308:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564175488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827658.975318:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827658.975322:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827658.975326:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564175488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827658.975351:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564175616:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827658.975374:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827658.975387:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827658/real 1787827658] req@ffff976889efe580 x1874670564175616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827658.975400:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564175616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827658.975409:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827658.975413:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827658.975417:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564175616:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827658.975424:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564175744:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827658.975444:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827658.975456:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827658/real 1787827658] req@ffff976889efe1c0 x1874670564175744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827658.975469:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564175744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827658.975484:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827658.975488:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827658.975491:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564175744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827658.975497:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564175872:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827658.975517:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827658.975529:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827658/real 1787827658] req@ffff976889efcb40 x1874670564175872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827658.975541:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564175872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827658.975550:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827658.975553:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827658.975557:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564175872:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827659.230924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827659.230924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827659.230987:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827659.998881:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827660.254966:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827660.254967:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827660.254987:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827661.022925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827661.278968:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827661.278973:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827661.279078:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827662.046884:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827662.302901:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827662.302912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827662.302972:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827663.070924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827663.326915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827663.326915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827663.326969:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827664.094942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827664.094953:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827664.094960:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827664.094966:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827664.094976:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2603 00000100:00080000:3.0:1787827664.094989:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827664.094997:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827664.095041:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564176000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827664.095064:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827664.095067:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827664.095068:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827664.095072:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2603 00000100:00080000:3.0:1787827664.095075:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827664.095078:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827664.095089:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827664.095100:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564176128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827664.095109:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827664.095111:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827664.095113:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827664.095117:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2603 00000100:00080000:3.0:1787827664.095120:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827664.095121:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564176000:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827664.095123:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827664.095130:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827664.095138:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564176256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827664.095144:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827664.095146:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827664.095155:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827664.095158:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2603 00000100:00080000:3.0:1787827664.095161:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827664.095164:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827664.095168:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827664.095176:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564176384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827664.095188:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827664.095190:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827664.095192:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827664.095194:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2603 00000100:00080000:3.0:1787827664.095197:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827664.095199:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827664.095202:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827664.095212:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564176512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827664.095214:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827664.095294:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827664/real 1787827664] req@ffff97688cb5de00 x1874670564176000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827664.095317:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564176000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827664.095331:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827664.095335:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827664.095340:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564176000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827664.095371:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564176128:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827664.095395:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827664.095409:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827664/real 1787827664] req@ffff97688cb5cb40 x1874670564176128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827664.095423:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564176128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827664.095432:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827664.095435:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827664.095439:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564176128:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827664.095445:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564176256:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827664.095463:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827664.095473:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827664/real 1787827664] req@ffff97688cb5c000 x1874670564176256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827664.095486:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564176256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827664.095495:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827664.095498:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827664.095502:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564176256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827664.095509:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564176384:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827664.095527:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827664.095538:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827664/real 1787827664] req@ffff97688cb5d680 x1874670564176384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827664.095550:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564176384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827664.095558:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827664.095562:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827664.095566:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564176384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827664.095572:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564176512:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827664.095609:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827664.095621:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827664/real 1787827664] req@ffff97688cb5f0c0 x1874670564176512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827664.095633:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564176512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827664.095642:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827664.095646:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827664.095650:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564176512:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827664.350916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827664.350947:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827664.350983:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827665.118956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827665.374951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827665.374954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827665.374960:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827666.142883:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827666.398950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827666.398950:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827666.399020:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827667.166975:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827667.422978:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827667.422978:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827667.423058:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827668.190904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827668.446948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827668.446950:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827668.447021:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827669.214903:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827669.214909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827669.214914:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827669.214920:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827669.214931:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2609 00000100:00080000:3.0:1787827669.214943:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827669.214951:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827669.215003:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564176640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827669.215028:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827669.215031:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827669.215033:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827669.215036:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2609 00000100:00080000:3.0:1787827669.215039:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827669.215041:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827669.215052:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827669.215063:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564176768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827669.215072:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827669.215075:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827669.215076:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827669.215079:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2609 00000100:00080000:3.0:1787827669.215082:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827669.215085:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827669.215088:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827669.215095:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564176896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827669.215101:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827669.215105:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827669.215106:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827669.215110:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2609 00000100:00080000:3.0:1787827669.215112:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827669.215115:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827669.215119:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827669.215127:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564176640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827669.215131:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564177024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827669.215149:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827669.215151:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827669.215153:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827669.215158:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2609 00000100:00080000:3.0:1787827669.215161:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827669.215164:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827669.215167:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827669.215178:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564177152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827669.215209:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827669.215248:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827669/real 1787827669] req@ffff97688cb5f840 x1874670564176640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827669.215261:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564176640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827669.215271:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827669.215273:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827669.215277:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564176640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827669.215282:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564176768:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827669.215297:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827669.215305:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827669/real 1787827669] req@ffff97688cb5c3c0 x1874670564176768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827669.215312:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564176768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827669.215318:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827669.215320:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827669.215322:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564176768:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827669.215326:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564176896:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827669.215344:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827669.215350:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827669/real 1787827669] req@ffff97688cb5e1c0 x1874670564176896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827669.215358:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564176896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827669.215364:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827669.215366:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827669.215368:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564176896:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827669.215393:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564177024:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827669.215404:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827669.215410:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827669/real 1787827669] req@ffff97688cb5c780 x1874670564177024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827669.215418:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564177024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827669.215423:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827669.215425:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827669.215427:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564177024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827669.215430:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564177152:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827669.215440:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827669.215446:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827669/real 1787827669] req@ffff97688cb5cf00 x1874670564177152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827669.215453:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564177152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827669.215462:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827669.215465:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827669.215467:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564177152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827669.470947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827669.470949:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827669.471068:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827670.238946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827670.494918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827670.494964:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827670.495039:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827671.262955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827671.518927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827671.518961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827671.518991:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827672.286978:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827672.542933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827672.542934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827672.542983:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827673.310910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827673.566931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827673.566931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827673.567038:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827674.334957:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827674.334958:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827674.334970:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827674.334976:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827674.334986:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2614 00000100:00080000:3.0:1787827674.335000:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827674.335009:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827674.335063:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564177280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827674.335103:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827674.335107:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827674.335108:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827674.335113:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2614 00000100:00080000:3.0:1787827674.335116:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827674.335120:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827674.335130:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827674.335144:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564177408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827674.335158:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827674.335161:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827674.335162:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827674.335166:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2614 00000100:00080000:3.0:1787827674.335168:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827674.335170:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827674.335173:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827674.335178:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564177280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827674.335183:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564177536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827674.335194:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827674.335196:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827674.335197:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827674.335200:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2614 00000100:00080000:3.0:1787827674.335203:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827674.335205:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827674.335209:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827674.335217:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564177664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827674.335222:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827674.335225:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827674.335226:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827674.335229:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2614 00000100:00080000:3.0:1787827674.335232:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827674.335235:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827674.335239:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827674.335246:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564177792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827674.335270:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827674.335331:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827674/real 1787827674] req@ffff97688cb5d2c0 x1874670564177280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827674.335353:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564177280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827674.335365:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827674.335371:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827674.335375:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564177280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827674.335383:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564177408:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827674.335409:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827674.335424:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827674/real 1787827674] req@ffff97688cb5cf00 x1874670564177408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827674.335437:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564177408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827674.335446:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827674.335450:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827674.335453:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564177408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827674.335486:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564177536:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827674.335525:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827674.335538:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827674/real 1787827674] req@ffff97688cb5c780 x1874670564177536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827674.335550:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564177536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827674.335559:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827674.335562:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827674.335565:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564177536:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827674.335572:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564177664:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827674.335592:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827674.335604:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827674/real 1787827674] req@ffff97688cb5e1c0 x1874670564177664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827674.335617:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564177664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827674.335625:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827674.335628:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827674.335632:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564177664:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827674.335638:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564177792:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827674.335657:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827674.335669:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827674/real 1787827674] req@ffff97688cb5c3c0 x1874670564177792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827674.335681:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564177792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827674.335689:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827674.335693:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827674.335696:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564177792:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827674.590919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827674.590919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827674.591123:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827675.358915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827675.614961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827675.614963:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827675.615007:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827676.382947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827676.638969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827676.638969:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827676.639069:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827677.406947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827677.662908:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827677.663008:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827677.663023:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827678.430982:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827678.686976:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827678.686976:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827678.687042:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827679.454910:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827679.454919:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827679.454923:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827679.454932:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2619 00000100:00080000:3.0:1787827679.454943:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827679.454950:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827679.454975:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787827679.454991:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564177920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827679.455018:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827679.455022:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827679.455029:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827679.455033:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2619 00000100:00080000:3.0:1787827679.455035:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827679.455039:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827679.455048:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827679.455058:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564178048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827679.455067:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827679.455073:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827679.455074:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827679.455081:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2619 00000100:00100000:0.0:1787827679.455082:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564177920:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827679.455084:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827679.455091:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827679.455096:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827679.455103:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564178176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827679.455110:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827679.455112:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827679.455114:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827679.455117:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2619 00000100:00080000:3.0:1787827679.455120:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827679.455123:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827679.455126:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827679.455134:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564178304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827679.455141:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827679.455144:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827679.455145:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827679.455147:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2619 00000100:00080000:3.0:1787827679.455150:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827679.455153:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827679.455158:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787827679.455166:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827679.455170:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564178432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827679.455217:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827679/real 1787827679] req@ffff97688cb5f840 x1874670564177920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827679.455234:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564177920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827679.455244:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827679.455247:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827679.455251:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564177920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827679.455257:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564178048:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827679.455280:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827679.455290:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827679/real 1787827679] req@ffff97688cb5f0c0 x1874670564178048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827679.455300:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564178048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827679.455306:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827679.455308:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827679.455311:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564178048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827679.455332:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564178176:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827679.455345:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827679.455352:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827679/real 1787827679] req@ffff97688cb5d680 x1874670564178176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827679.455360:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564178176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827679.455366:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827679.455369:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827679.455388:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564178176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827679.455392:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564178304:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827679.455406:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827679.455415:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827679/real 1787827679] req@ffff97688cb5c000 x1874670564178304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827679.455424:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564178304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827679.455430:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827679.455432:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827679.455434:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564178304:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827679.455438:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564178432:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827679.455451:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827679.455459:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827679/real 1787827679] req@ffff97688cb5cb40 x1874670564178432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827679.455487:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564178432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827679.455496:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827679.455498:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827679.455502:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564178432:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827679.711048:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827679.711075:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827679.711179:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827680.478985:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827680.735002:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827680.735062:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827680.735086:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827681.503097:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827681.758890:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827681.758924:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827681.758948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827682.527041:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827682.782930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827682.782955:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827682.782994:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827683.550958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827683.806901:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827683.806939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827683.806958:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827684.574942:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827684.574959:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1787827684.574961:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827684.574968:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827684.574989:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2624 00000100:00080000:3.0:1787827684.575007:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827684.575019:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827684.575075:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564178560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827684.575119:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827684.575124:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827684.575127:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827684.575133:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2624 00000100:00080000:3.0:1787827684.575138:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827684.575143:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827684.575155:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827684.575175:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564178688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827684.575191:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827684.575195:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827684.575198:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827684.575209:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564178560:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827684.575209:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2624 00000100:00080000:3.0:1787827684.575220:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827684.575240:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827684.575248:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827684.575265:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564178816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827684.575279:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827684.575284:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827684.575286:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827684.575292:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2624 00000100:00080000:3.0:1787827684.575300:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827684.575307:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827684.575317:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1787827684.575321:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827684.575352:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564178944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827684.575377:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827684.575384:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827684.575387:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827684.575396:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2624 00000100:00080000:3.0:1787827684.575404:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827684.575410:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827684.575422:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827684/real 1787827684] req@ffff97688cb5de00 x1874670564178560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827684.575422:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827684.575481:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564179072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827684.575570:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564178560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827684.575589:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827684.575596:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827684.575602:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564178560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827684.575643:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564178688:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827684.575689:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827684.575713:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827684/real 1787827684] req@ffff97688cb5cb40 x1874670564178688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827684.575732:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564178688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827684.575747:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827684.575752:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827684.575757:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564178688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827684.575769:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564178816:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827684.575803:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827684.575897:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827684/real 1787827684] req@ffff97688cb5c000 x1874670564178816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827684.575924:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564178816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827684.575944:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827684.575949:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827684.575953:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564178816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827684.575962:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564178944:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827684.575987:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827684.576003:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827684/real 1787827684] req@ffff97688cb5d680 x1874670564178944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827684.576020:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564178944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827684.576031:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827684.576035:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827684.576040:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564178944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827684.576049:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564179072:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827684.576068:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827684.576079:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827684/real 1787827684] req@ffff97688cb5f0c0 x1874670564179072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827684.576092:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564179072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827684.576103:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827684.576108:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827684.576112:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564179072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827684.830953:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827684.831023:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827684.831089:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827685.598930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827685.854978:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827685.855023:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827685.855033:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827686.622992:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827686.878949:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827686.878953:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827686.879018:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827687.647013:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827687.903020:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827687.903020:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827687.903035:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827688.670920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827688.926975:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827688.926975:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827688.927070:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827689.694922:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827689.694922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827689.694934:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827689.694939:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827689.694951:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2629 00000100:00080000:0.0:1787827689.694968:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827689.694976:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827689.695007:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564179200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827689.695033:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827689.695053:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827689.695055:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827689.695059:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2629 00000100:00080000:0.0:1787827689.695062:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827689.695065:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827689.695075:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827689.695090:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564179328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827689.695091:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564179200:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787827689.695116:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827689.695119:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827689.695121:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827689.695126:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2629 00000100:00080000:0.0:1787827689.695128:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827689.695132:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827689.695135:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827689.695160:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564179456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827689.695169:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827689.695171:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827689.695172:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827689.695176:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2629 00000100:00080000:0.0:1787827689.695178:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827689.695187:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827689.695191:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827689.695199:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564179584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827689.695206:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787827689.695215:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787827689.695217:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827689.695223:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827689.695230:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2629 00000100:00080000:0.0:1787827689.695234:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827689.695238:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827689.695245:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827689.695259:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564179712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827689.695278:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827689/real 1787827689] req@ffff976882aef0c0 x1874670564179200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827689.695303:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564179200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827689.695319:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827689.695324:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827689.695327:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564179200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827689.695359:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564179328:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827689.695416:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827689.695433:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827689/real 1787827689] req@ffff976882aee940 x1874670564179328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827689.695443:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564179328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827689.695450:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827689.695452:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827689.695455:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564179328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827689.695460:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564179456:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827689.695484:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827689.695493:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827689/real 1787827689] req@ffff976882aee580 x1874670564179456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827689.695501:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564179456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827689.695507:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827689.695510:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827689.695512:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564179456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827689.695517:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564179584:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827689.695531:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827689.695537:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827689/real 1787827689] req@ffff976882aec3c0 x1874670564179584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827689.695545:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564179584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827689.695551:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827689.695553:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827689.695555:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564179584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827689.695559:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564179712:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827689.695570:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827689.695575:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827689/real 1787827689] req@ffff976882aeed00 x1874670564179712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827689.695583:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564179712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827689.695588:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827689.695590:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827689.695591:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564179712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827689.950903:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827689.950930:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827689.950950:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827690.718960:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827690.974982:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827690.974991:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827690.975023:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827691.742947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827691.998905:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827691.998933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827691.998945:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827692.766912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827693.022877:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827693.022901:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827693.022979:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827693.790990:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827694.046949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827694.046959:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827694.046986:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827694.814934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827694.814966:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827694.814975:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827694.814981:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827694.815008:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2634 00000100:00080000:0.0:1787827694.815033:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827694.815041:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827694.815078:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564179840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827694.815103:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827694.815107:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827694.815109:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827694.815112:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2634 00000100:00080000:0.0:1787827694.815116:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827694.815120:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827694.815160:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827694.815170:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564179840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827694.815198:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564179968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827694.815244:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827694.815248:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827694.815251:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827694.815255:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2634 00000100:00080000:0.0:1787827694.815260:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827694.815266:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787827694.815287:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827694.815294:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827694.815308:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564180096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827694.815317:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827694.815320:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827694.815322:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827694.815326:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2634 00000100:00080000:0.0:1787827694.815331:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827694.815332:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827694/real 1787827694] req@ffff976882aed680 x1874670564179840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827694.815338:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827694.815348:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827694.815360:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564179840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827694.815396:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564180224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827694.815407:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827694.815410:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827694.815414:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:0.0:1787827694.815417:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827694.815421:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827694.815426:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564179840:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787827694.815429:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2634 00000100:00080000:0.0:1787827694.815449:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827694.815453:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827694.815459:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827694.815475:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564180352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827694.815495:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564179968:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827694.815531:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827694.815550:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827694/real 1787827694] req@ffff976882aeed00 x1874670564179968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827694.815562:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564179968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827694.815571:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827694.815575:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827694.815580:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564179968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827694.815589:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564180096:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827694.815622:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827694.815631:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827694/real 1787827694] req@ffff976882aec3c0 x1874670564180096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827694.815640:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564180096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827694.815646:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827694.815648:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827694.815650:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564180096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827694.815654:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564180224:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827694.815667:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827694.815674:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827694/real 1787827694] req@ffff976882aee580 x1874670564180224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827694.815682:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564180224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827694.815688:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827694.815691:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827694.815693:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564180224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827694.815717:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564180352:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827694.815731:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827694.815739:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827694/real 1787827694] req@ffff976882aee940 x1874670564180352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827694.815748:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564180352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827694.815754:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827694.815757:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827694.815759:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564180352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827695.070869:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827695.070898:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827695.070941:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827695.838978:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827696.094891:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827696.094906:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827696.094912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827696.862903:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827697.118883:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827697.118888:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827697.118952:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827697.886909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827698.142923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827698.142923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827698.142945:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827698.910893:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827699.166892:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827699.166902:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827699.166985:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827699.934936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827699.934962:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827699.934970:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827699.934975:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827699.934985:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2639 00000100:00080000:3.0:1787827699.934998:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827699.935006:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827699.935045:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564180480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827699.935074:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827699.935078:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827699.935080:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827699.935086:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2639 00000100:00080000:3.0:1787827699.935091:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827699.935097:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827699.935104:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564180480:192.168.203.139@tcp:250:kworker.0 00000008:00100000:3.0:1787827699.935107:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827699.935125:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564180608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827699.935137:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827699.935141:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827699.935143:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827699.935147:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2639 00000100:00080000:3.0:1787827699.935152:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827699.935156:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:2.0:1787827699.935160:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:3.0:1787827699.935164:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827699.935180:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564180736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827699.935191:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827699.935195:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827699.935197:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827699.935202:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2639 00000100:00100000:2.0:1787827699.935205:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827699/real 1787827699] req@ffff97688cb5f840 x1874670564180480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827699.935208:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827699.935216:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827699.935223:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827699.935228:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564180480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827699.935238:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787827699.935241:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564180864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827699.935260:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827699.935267:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564180480:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827699.935279:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827699.935284:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827699.935286:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827699.935292:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2639 00000100:00080000:3.0:1787827699.935296:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827699.935298:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564180608:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787827699.935304:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827699.935311:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827699.935323:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827699.935345:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564180992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827699.935354:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827699/real 1787827699] req@ffff97688cb5c3c0 x1874670564180608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827699.935389:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564180608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827699.935395:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827699.935398:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827699.935400:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564180608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827699.935406:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564180736:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827699.935422:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827699.935429:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827699/real 1787827699] req@ffff97688cb5e1c0 x1874670564180736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827699.935436:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564180736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827699.935441:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827699.935444:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827699.935446:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564180736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827699.935449:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564180864:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827699.935459:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827699.935465:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827699/real 1787827699] req@ffff97688cb5c780 x1874670564180864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827699.935472:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564180864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827699.935477:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827699.935479:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827699.935481:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564180864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827699.935493:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564180992:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827699.935503:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827699.935509:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827699/real 1787827699] req@ffff97688cb5cf00 x1874670564180992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827699.935516:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564180992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827699.935521:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827699.935523:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827699.935525:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564180992:192.168.203.139@tcp:8:kworker.0 00010000:00010000:3.0:1787827700.190915:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827700.190917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827700.190917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827700.958920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827701.215004:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827701.215005:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827701.215029:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827701.982906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827702.238966:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827702.238990:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827702.239011:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827703.006915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827703.262958:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827703.262970:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827703.263037:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827704.030940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827704.286954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827704.286998:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827704.287075:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827705.054959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827705.055021:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827705.055032:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827705.055038:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827705.055079:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2644 00000100:00080000:3.0:1787827705.055092:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827705.055110:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827705.055166:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564181120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827705.055199:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827705.055203:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827705.055206:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827705.055211:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2644 00000100:00080000:3.0:1787827705.055215:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827705.055220:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827705.055250:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827705.055270:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564181248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827705.055277:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564181120:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827705.055348:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827705.055354:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827705.055357:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827705.055363:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2644 00000100:00080000:3.0:1787827705.055399:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827705.055408:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827705.055419:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827705.055439:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827705.055450:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564181376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827705.055475:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827705.055482:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827705.055486:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827705.055495:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827705/real 1787827705] req@ffff97688cb5d2c0 x1874670564181120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827705.055501:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2644 00000100:00080000:3.0:1787827705.055514:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827705.055525:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827705.055535:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827705.055536:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564181120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827705.055556:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787827705.055563:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564181504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827705.055568:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827705.055582:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564181120:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827705.055602:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827705.055607:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827705.055609:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827705.055615:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2644 00000100:00080000:3.0:1787827705.055621:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827705.055631:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564181248:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787827705.055632:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827705.055648:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827705.055669:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564181632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827705.055729:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827705.055758:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827705/real 1787827705] req@ffff97688cb5cf00 x1874670564181248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827705.055783:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564181248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827705.055819:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827705.055826:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827705.055833:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564181248:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827705.055879:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564181376:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827705.055914:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827705.055931:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827705/real 1787827705] req@ffff97688cb5c780 x1874670564181376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827705.055947:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564181376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827705.055958:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827705.055962:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827705.055967:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564181376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827705.055978:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564181504:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827705.056015:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827705.056038:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827705/real 1787827705] req@ffff97688cb5e1c0 x1874670564181504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827705.056078:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564181504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827705.056094:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827705.056098:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827705.056103:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564181504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827705.056151:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564181632:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827705.056191:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827705.056209:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827705/real 1787827705] req@ffff97688cb5c3c0 x1874670564181632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827705.056241:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564181632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827705.056253:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827705.056257:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827705.056261:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564181632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827705.310940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827705.310949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827705.311072:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827706.078919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827706.334940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827706.334969:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827706.335017:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827707.102916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827707.358914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827707.358930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827707.359087:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827708.126929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827708.382983:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827708.383036:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827708.383048:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827709.150909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827709.406919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827709.407014:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827709.407150:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827710.174902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827710.174989:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827710.174994:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827710.174998:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827710.175006:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2650 00000100:00080000:3.0:1787827710.175015:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827710.175021:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827710.175059:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564181760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827710.175129:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564181760:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827710.175289:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827710.175294:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827710.175297:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827710.175320:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2650 00000100:00080000:3.0:1787827710.175325:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827710.175329:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827710.175338:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827710.175352:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564181888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827710.175388:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564181888:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787827710.175475:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827710.175479:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827710.175484:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827710.175487:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2650 00000100:00080000:3.0:1787827710.175491:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827710.175494:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827710.175499:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827710.175511:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564182016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827710.175524:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827710.175528:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827710.175530:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827710.175538:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564182016:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787827710.175538:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2650 00000100:00080000:3.0:1787827710.175544:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827710.175551:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827710.175564:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827710.175581:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564182144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827710.175598:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827710.175602:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827710.175605:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827710.175610:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2650 00000100:00080000:3.0:1787827710.175615:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827710.175620:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564182144:192.168.203.139@tcp:8:kworker.0 00000100:00080000:3.0:1787827710.175623:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827710.175636:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827710.175652:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564182272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827710.175682:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564182272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827710.430962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827710.431016:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827710.431105:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827711.198937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827711.454942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827711.454972:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827711.455103:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827712.222877:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827712.478930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827712.478974:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827712.479043:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827713.246910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:0.0:1787827713.247099:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:0.0:1787827713.247138:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827713.247154:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787827713.247193:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787827713.247210:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:2.0:1787827713.247212:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827710/real 1787827713] req@ffff97688cb5f840 x1874670564181760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787827713.247229:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:2.0:1787827713.247249:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564181760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787827713.247260:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00080000:2.0:1787827713.247269:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827713.247278:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827713.247283:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564181760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827713.247308:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827710/real 1787827713] req@ffff97688cb5cb40 x1874670564182272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827713.247325:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564182272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827713.247335:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827713.247339:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827713.247343:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564182272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827713.247350:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827710/real 1787827713] req@ffff97688cb5c000 x1874670564182144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827713.247362:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564182144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827713.247369:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827713.247372:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827713.247375:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564182144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827713.247381:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827710/real 1787827713] req@ffff97688cb5d680 x1874670564182016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827713.247391:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564182016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827713.247399:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827713.247402:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827713.247406:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564182016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827713.247411:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827710/real 1787827713] req@ffff97688cb5f0c0 x1874670564181888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827713.247424:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564181888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827713.247432:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827713.247435:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827713.247439:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564181888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827713.502883:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827713.502889:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827713.502970:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827714.270989:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827714.526964:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827714.526987:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827714.527130:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827715.294958:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827715.294960:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827715.294971:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827715.294978:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827715.294989:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2655 00000100:00080000:2.0:1787827715.295001:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827715.295020:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827715.295065:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564182400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827715.295104:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827715.295108:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827715.295110:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827715.295114:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2655 00000100:00080000:2.0:1787827715.295119:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827715.295123:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827715.295134:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827715.295154:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564182528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827715.295170:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564182400:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787827715.295175:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827715.295186:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827715.295189:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827715.295196:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2655 00000100:00080000:2.0:1787827715.295203:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827715.295210:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827715.295238:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827715.295261:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564182656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827715.295281:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787827715.295290:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827715.295297:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827715.295301:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827715.295310:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2655 00000100:00080000:2.0:1787827715.295318:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827715.295326:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827715.295333:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827715/real 1787827715] req@ffff976889cfad00 x1874670564182400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827715.295338:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827715.295362:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564182400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827715.295369:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564182784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827715.295400:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827715.295404:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827715.295412:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:2.0:1787827715.295414:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827715.295423:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827715.295427:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564182400:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787827715.295434:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2655 00000100:00080000:2.0:1787827715.295445:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827715.295453:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827715.295462:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827715.295472:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564182528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827715.295484:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564182912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827715.295529:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827715.295558:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827715/real 1787827715] req@ffff976889cf92c0 x1874670564182528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827715.295590:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564182528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827715.295603:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827715.295608:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827715.295612:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564182528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827715.295619:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564182656:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827715.295652:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827715.295670:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827715/real 1787827715] req@ffff976889cfa1c0 x1874670564182656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827715.295689:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564182656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827715.295701:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827715.295706:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827715.295710:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564182656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827715.295729:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564182784:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827715.295783:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827715.295803:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827715/real 1787827715] req@ffff976889cfa580 x1874670564182784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827715.295821:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564182784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827715.295876:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827715.295882:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827715.295888:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564182784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827715.295924:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564182912:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827715.295957:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827715.295977:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827715/real 1787827715] req@ffff976889cfb480 x1874670564182912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827715.295996:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564182912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827715.296009:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827715.296014:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827715.296019:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564182912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827715.550939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827715.550949:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827715.550960:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827716.318946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827716.574953:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827716.574984:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827716.575001:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827717.342953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827717.598900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827717.598900:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827717.598950:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827718.366915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827718.622898:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827718.622922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827718.622967:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827719.390923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827719.646968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827719.646971:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827719.647037:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827720.414917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827720.414921:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827720.414931:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827720.414936:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827720.414948:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2660 00000100:00080000:2.0:1787827720.414962:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827720.414970:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827720.415030:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564183040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827720.415059:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827720.415062:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827720.415064:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827720.415068:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2660 00000100:00080000:2.0:1787827720.415070:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827720.415073:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827720.415095:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827720.415105:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564183168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827720.415114:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827720.415118:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827720.415120:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827720.415122:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2660 00000100:00080000:2.0:1787827720.415125:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827720.415128:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827720.415132:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827720.415139:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564183040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827720.415144:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564183296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827720.415162:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827720.415165:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827720.415166:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827720.415169:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2660 00000100:00080000:2.0:1787827720.415171:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827720.415174:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827720.415177:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827720.415200:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564183424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827720.415207:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827720.415210:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827720.415212:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827720.415215:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2660 00000100:00080000:2.0:1787827720.415217:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827720.415219:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827720.415223:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827720.415231:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564183552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827720.415255:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827720.415312:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827720/real 1787827720] req@ffff976889cf9e00 x1874670564183040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827720.415333:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564183040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827720.415343:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827720.415347:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827720.415352:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564183040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827720.415359:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564183168:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827720.415383:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827720.415393:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827720/real 1787827720] req@ffff976889cfa940 x1874670564183168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827720.415402:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564183168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827720.415410:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827720.415413:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827720.415416:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564183168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827720.415442:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564183296:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827720.415459:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827720.415468:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827720/real 1787827720] req@ffff976889cfbc00 x1874670564183296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827720.415478:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564183296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827720.415492:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827720.415497:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827720.415500:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564183296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827720.415506:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564183424:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827720.415520:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827720.415531:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827720/real 1787827720] req@ffff976889cf8000 x1874670564183424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827720.415542:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564183424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827720.415550:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827720.415552:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827720.415556:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564183424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827720.415561:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564183552:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827720.415586:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827720.415594:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827720/real 1787827720] req@ffff976889cfb0c0 x1874670564183552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827720.415606:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564183552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827720.415612:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827720.415614:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827720.415617:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564183552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827720.670936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827720.670937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827720.671001:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827721.438908:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827721.694945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827721.694963:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827721.694989:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827722.462944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827722.718905:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827722.718915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827722.718973:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827723.486919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827723.742975:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827723.742976:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827723.743017:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827724.510897:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827724.766927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827724.766928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827724.767031:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827725.534956:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827725.534959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827725.534968:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827725.534977:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827725.534990:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2665 00000100:00080000:2.0:1787827725.535005:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827725.535013:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827725.535060:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564183680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827725.535090:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827725.535093:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827725.535094:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827725.535098:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2665 00000100:00080000:2.0:1787827725.535100:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827725.535102:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827725.535113:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827725.535129:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564183808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827725.535138:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827725.535141:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827725.535143:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827725.535146:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2665 00000100:00080000:2.0:1787827725.535164:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827725.535169:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827725.535173:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827725.535186:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564183936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827725.535196:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827725.535199:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827725.535200:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827725.535204:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2665 00000100:00080000:2.0:1787827725.535206:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827725.535210:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827725.535214:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827725.535225:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564184064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827725.535228:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564183680:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787827725.535244:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827725.535251:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827725.535253:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827725.535256:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2665 00000100:00080000:2.0:1787827725.535258:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827725.535260:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827725.535264:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827725.535273:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564184192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827725.535364:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827725.535427:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827725/real 1787827725] req@ffff976889cf8f00 x1874670564183680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827725.535460:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564183680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827725.535481:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827725.535487:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827725.535494:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564183680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827725.535509:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564183808:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827725.535553:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827725.535590:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827725/real 1787827725] req@ffff976889cfb0c0 x1874670564183808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827725.535615:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564183808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827725.535631:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827725.535637:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827725.535644:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564183808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827725.535681:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564183936:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827725.535721:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827725.535764:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827725/real 1787827725] req@ffff976889cf8000 x1874670564183936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827725.535791:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564183936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827725.535809:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827725.535816:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827725.535824:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564183936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827725.535837:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564184064:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827725.535932:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827725.535957:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827725/real 1787827725] req@ffff976889cfbc00 x1874670564184064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827725.535985:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564184064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827725.536002:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827725.536009:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827725.536025:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564184064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827725.536037:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564184192:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827725.536078:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827725.536104:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827725/real 1787827725] req@ffff976889cfa940 x1874670564184192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827725.536129:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564184192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827725.536146:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827725.536195:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827725.536203:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564184192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827725.790912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827725.790937:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827725.790944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827726.558905:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827726.814902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827726.814909:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827726.814940:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827727.582950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827727.838967:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827727.838977:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827727.838989:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827728.606917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827728.862905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827728.862908:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827728.862936:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827729.630913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827729.886905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827729.886930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827729.886964:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827730.654940:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827730.654948:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827730.654952:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827730.654957:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827730.654960:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2670 00000100:00080000:2.0:1787827730.654973:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827730.654982:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827730.655017:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564184320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827730.655042:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827730.655045:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827730.655046:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827730.655049:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2670 00000100:00080000:2.0:1787827730.655052:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827730.655054:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827730.655062:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827730.655072:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564184448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827730.655081:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827730.655084:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827730.655085:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827730.655089:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2670 00000100:00080000:2.0:1787827730.655091:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827730.655093:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827730.655099:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827730.655106:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564184320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827730.655107:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564184576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827730.655135:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827730.655137:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827730.655138:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827730.655144:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2670 00000100:00080000:2.0:1787827730.655146:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827730.655151:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827730.655154:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827730.655163:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564184704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827730.655169:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827730.655173:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827730.655174:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827730.655176:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2670 00000100:00080000:2.0:1787827730.655180:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827730.655183:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827730.655186:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827730.655192:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564184832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827730.655210:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827730.655273:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827730/real 1787827730] req@ffff976889cf9e00 x1874670564184320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827730.655289:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564184320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827730.655300:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827730.655304:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827730.655308:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564184320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827730.655315:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564184448:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827730.655340:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827730.655350:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827730/real 1787827730] req@ffff976889cfb480 x1874670564184448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827730.655358:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564184448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827730.655364:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827730.655367:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827730.655369:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564184448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827730.655394:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564184576:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827730.655405:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827730.655412:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827730/real 1787827730] req@ffff976889cfa580 x1874670564184576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827730.655420:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564184576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827730.655426:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827730.655428:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827730.655430:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564184576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827730.655434:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564184704:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827730.655445:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827730.655451:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827730/real 1787827730] req@ffff976889cfa1c0 x1874670564184704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827730.655463:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564184704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827730.655468:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827730.655470:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827730.655472:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564184704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827730.655476:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564184832:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827730.655486:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827730.655492:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827730/real 1787827730] req@ffff976889cf92c0 x1874670564184832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827730.655499:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564184832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827730.655505:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827730.655506:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827730.655508:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564184832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827730.910910:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827730.910942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827730.910971:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827731.678926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827731.934928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827731.934937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827731.934988:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827732.702938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827732.958929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827732.958929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827732.958999:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827733.726893:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827733.982902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827733.982906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827733.982958:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827734.751014:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827735.006951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827735.006952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827735.006990:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827735.774939:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827735.774951:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1787827735.774955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827735.774957:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827735.774984:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2675 00000100:00080000:1.0:1787827735.774999:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827735.775008:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827735.775056:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564184960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827735.775129:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827735.775133:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827735.775134:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827735.775138:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2675 00000100:00080000:1.0:1787827735.775141:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827735.775144:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827735.775155:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827735.775167:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564185088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827735.775175:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827735.775178:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827735.775179:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827735.775182:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2675 00000100:00080000:1.0:1787827735.775186:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827735.775192:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564184960:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827735.775192:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827735.775197:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827735.775208:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564185216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827735.775215:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827735.775217:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827735.775218:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827735.775221:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2675 00000100:00080000:1.0:1787827735.775223:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827735.775227:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827735.775233:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827735.775247:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564185344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827735.775258:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827735.775262:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827735.775264:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827735.775268:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2675 00000100:00080000:1.0:1787827735.775273:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787827735.775276:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787827735.775281:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827735.775286:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827735.775298:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564185472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827735.775337:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827735/real 1787827735] req@ffff976889eff0c0 x1874670564184960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827735.775358:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564184960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827735.775371:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827735.775376:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827735.775381:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564184960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827735.775390:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564185088:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827735.775412:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827735.775423:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827735/real 1787827735] req@ffff976889efed00 x1874670564185088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827735.775432:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564185088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827735.775439:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827735.775441:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827735.775443:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564185088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827735.775466:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564185216:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827735.775479:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827735.775486:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827735/real 1787827735] req@ffff976889efc3c0 x1874670564185216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827735.775494:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564185216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827735.775504:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827735.775507:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827735.775509:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564185216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827735.775512:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564185344:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827735.775522:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827735.775528:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827735/real 1787827735] req@ffff976889efe940 x1874670564185344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827735.775535:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564185344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827735.775552:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827735.775555:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827735.775560:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564185344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827735.775564:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564185472:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827735.775576:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827735.775583:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827735/real 1787827735] req@ffff976889effc00 x1874670564185472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827735.775603:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564185472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827735.775608:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827735.775610:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827735.775613:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564185472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827736.030907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827736.030950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827736.030992:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827736.798978:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827737.054941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827737.054968:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827737.054977:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827737.822914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827738.078948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827738.078954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827738.079020:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827738.846911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827739.102939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827739.102948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827739.102977:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827739.871035:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827740.126950:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827740.126971:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827740.127064:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827740.894961:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827740.894967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827740.894973:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827740.894979:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827740.894989:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2680 00000100:00080000:1.0:1787827740.895017:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827740.895026:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827740.895108:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564185600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827740.895134:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827740.895137:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827740.895139:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827740.895143:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2680 00000100:00080000:1.0:1787827740.895146:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827740.895149:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827740.895159:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827740.895170:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564185728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827740.895198:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827740.895203:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827740.895206:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827740.895208:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564185600:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827740.895213:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2680 00000100:00080000:1.0:1787827740.895219:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827740.895231:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827740.895236:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827740.895249:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564185856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827740.895258:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827740.895261:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827740.895263:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827740.895266:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2680 00000100:00080000:1.0:1787827740.895284:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827740.895287:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827740.895292:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827740.895303:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564185984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827740.895310:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787827740.895319:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827740.895323:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827740.895325:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827740.895329:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2680 00000100:00080000:1.0:1787827740.895338:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827740.895341:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827740.895344:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827740.895354:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564186112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827740.895364:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827740/real 1787827740] req@ffff976889effc00 x1874670564185600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827740.895386:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564185600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827740.895397:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827740.895401:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827740.895405:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564185600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827740.895429:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564185728:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827740.895456:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827740.895491:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827740/real 1787827740] req@ffff976889efe940 x1874670564185728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827740.895509:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564185728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827740.895520:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827740.895525:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827740.895529:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564185728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827740.895535:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564185856:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827740.895556:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827740.895571:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827740/real 1787827740] req@ffff976889efc3c0 x1874670564185856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827740.895588:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564185856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827740.895600:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827740.895604:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827740.895609:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564185856:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827740.895628:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564185984:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827740.895649:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827740.895660:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827740/real 1787827740] req@ffff976889efed00 x1874670564185984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827740.895669:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564185984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827740.895676:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827740.895679:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827740.895682:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564185984:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827740.895686:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564186112:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827740.895700:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827740.895708:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827740/real 1787827740] req@ffff976889eff0c0 x1874670564186112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827740.895717:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564186112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827740.895723:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827740.895725:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827740.895727:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564186112:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827741.150947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827741.150947:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827741.151003:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827741.918904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827742.174922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827742.174926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827742.174998:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827742.942896:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827743.198904:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827743.198918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827743.198925:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827743.966882:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827744.222896:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827744.222901:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827744.222961:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827744.990982:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827745.246959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827745.246959:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827745.246991:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827746.014909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827746.014929:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827746.014935:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827746.014939:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827746.014947:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2685 00000100:00080000:0.0:1787827746.014957:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827746.014964:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827746.015000:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564186240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827746.015046:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827746.015049:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827746.015050:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827746.015054:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2685 00000100:00080000:0.0:1787827746.015057:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827746.015061:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827746.015070:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827746.015086:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564186368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827746.015086:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564186240:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787827746.015103:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827746.015108:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827746.015110:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827746.015113:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2685 00000100:00080000:0.0:1787827746.015116:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827746.015119:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827746.015124:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827746.015134:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564186496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827746.015141:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827746.015144:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787827746.015148:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787827746.015149:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827746.015157:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2685 00000100:00080000:0.0:1787827746.015161:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827746.015166:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827746.015172:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827746.015186:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564186624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827746.015197:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827746.015201:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827746/real 1787827746] req@ffff976882aef0c0 x1874670564186240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827746.015203:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827746.015208:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827746.015218:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2685 00000100:00080000:0.0:1787827746.015224:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827746.015225:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564186240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827746.015234:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:2.0:1787827746.015239:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787827746.015243:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827746.015247:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827746.015254:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564186240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827746.015261:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564186752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827746.015279:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564186368:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827746.015302:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827746.015312:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827746/real 1787827746] req@ffff976882aee940 x1874670564186368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827746.015328:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564186368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827746.015338:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827746.015341:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827746.015343:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564186368:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827746.015348:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564186496:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827746.015367:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827746.015378:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827746/real 1787827746] req@ffff976882aee580 x1874670564186496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827746.015390:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564186496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827746.015399:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827746.015402:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827746.015405:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564186496:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827746.015420:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564186624:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827746.015453:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827746.015465:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827746/real 1787827746] req@ffff976882aec3c0 x1874670564186624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827746.015478:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564186624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827746.015495:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827746.015497:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827746.015499:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564186624:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827746.015504:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564186752:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827746.015545:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827746.015558:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827746/real 1787827746] req@ffff976882aeed00 x1874670564186752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827746.015567:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564186752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827746.015590:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827746.015592:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827746.015594:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564186752:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827746.270918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827746.270918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827746.270996:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827747.038904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827747.294917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827747.294923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827747.295037:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827748.062968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827748.318933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827748.318970:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827748.319146:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827749.086955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827749.342968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827749.342979:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827749.343063:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827750.110967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827750.366898:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827750.366924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827750.367027:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827751.134928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827751.135017:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827751.135026:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827751.135031:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827751.135043:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2691 00000100:00080000:1.0:1787827751.135056:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827751.135063:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827751.135102:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564186880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827751.135130:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827751.135133:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827751.135135:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827751.135138:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2691 00000100:00080000:1.0:1787827751.135142:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827751.135145:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827751.135155:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827751.135167:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564187008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827751.135174:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564186880:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827751.135184:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827751.135189:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827751.135191:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827751.135195:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2691 00000100:00080000:1.0:1787827751.135198:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827751.135201:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827751.135205:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827751.135216:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564187136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827751.135224:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827751.135227:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827751.135228:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827751.135231:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2691 00000100:00080000:1.0:1787827751.135237:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827751.135240:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827751.135244:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1787827751.135249:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827751.135257:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564187264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827751.135267:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827751.135270:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827751.135271:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827751.135275:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2691 00000100:00080000:1.0:1787827751.135278:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827751.135281:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827751.135303:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827751.135315:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827751/real 1787827751] req@ffff976889efcb40 x1874670564186880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827751.135320:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564187392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827751.135382:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564186880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827751.135397:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827751.135402:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827751.135407:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564186880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827751.135431:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564187008:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827751.135467:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827751.135493:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827751/real 1787827751] req@ffff976889efe1c0 x1874670564187008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827751.135507:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564187008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827751.135517:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827751.135520:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827751.135523:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564187008:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827751.135530:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564187136:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787827751.135548:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827751.135565:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827751/real 1787827751] req@ffff976889efe580 x1874670564187136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827751.135576:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564187136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827751.135585:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827751.135588:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827751.135591:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564187136:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827751.135597:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564187264:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827751.135651:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827751.135666:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827751/real 1787827751] req@ffff976889eff480 x1874670564187264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827751.135683:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564187264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827751.135694:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827751.135699:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827751.135704:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564187264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827751.135712:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564187392:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787827751.135734:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827751.135744:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827751/real 1787827751] req@ffff976889efc780 x1874670564187392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827751.135754:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564187392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827751.135762:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827751.135764:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827751.135767:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564187392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827751.391015:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827751.391065:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827751.391080:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827752.158956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827752.414938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827752.414981:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827752.415000:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827753.182900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827753.438949:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827753.438975:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827753.438982:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827754.206932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827754.462924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827754.462924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827754.463009:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827755.230905:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827755.486930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827755.486957:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827755.486987:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787827756.254906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827756.254962:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827756.254974:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827756.254980:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827756.254999:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2696 00000100:00080000:0.0:1787827756.255017:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827756.255026:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827756.255071:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564187520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827756.255111:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827756.255124:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827756.255127:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827756.255139:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2696 00000100:00080000:0.0:1787827756.255144:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827756.255147:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564187520:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787827756.255153:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827756.255173:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827756.255190:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564187648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827756.255202:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827756.255206:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827756.255207:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827756.255212:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2696 00000100:00080000:0.0:1787827756.255214:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827756.255218:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827756.255225:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827756.255233:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564187776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827756.255246:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827756.255249:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827756.255252:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827756.255256:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2696 00000100:00080000:0.0:1787827756.255259:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827756.255262:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827756.255268:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827756.255276:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564187904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827756.255284:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787827756.255288:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787827756.255293:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827756.255298:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827756.255307:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2696 00000100:00080000:0.0:1787827756.255313:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827756.255319:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827756.255327:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827756.255335:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827756/real 1787827756] req@ffff976882aed680 x1874670564187520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827756.255349:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564188032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827756.255384:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564187520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827756.255397:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827756.255505:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827756.255510:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564187520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827756.255540:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564187648:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827756.255616:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827756.255636:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827756/real 1787827756] req@ffff976882aeed00 x1874670564187648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827756.255649:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564187648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827756.255659:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827756.255662:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827756.255667:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564187648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827756.255674:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564187776:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827756.255692:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827756.255701:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827756/real 1787827756] req@ffff976882aec3c0 x1874670564187776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827756.255713:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564187776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827756.255722:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827756.255725:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827756.255729:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564187776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827756.255735:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564187904:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827756.255750:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827756.255759:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827756/real 1787827756] req@ffff976882aee580 x1874670564187904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827756.255770:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564187904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827756.255778:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827756.255781:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827756.255785:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564187904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827756.255792:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564188032:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827756.255808:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827756.255819:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827756/real 1787827756] req@ffff976882aee940 x1874670564188032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827756.255862:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564188032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827756.255872:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827756.255876:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827756.255880:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564188032:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827756.510939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827756.510944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827756.511001:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827757.278907:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827757.534984:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827757.534986:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827757.534989:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827758.302902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827758.558889:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827758.558939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827758.558962:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827759.326908:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827759.582876:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827759.582938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827759.582940:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827760.350897:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827760.606894:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827760.606930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827760.606946:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827761.374925:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827761.374937:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827761.374957:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827761.374958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827761.374969:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2701 00000100:00080000:0.0:1787827761.374981:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827761.374988:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827761.375054:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564188160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827761.375079:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827761.375083:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827761.375085:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827761.375088:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2701 00000100:00080000:0.0:1787827761.375091:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827761.375095:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827761.375107:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827761.375128:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564188160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827761.375137:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564188288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827761.375151:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827761.375155:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827761.375157:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827761.375162:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2701 00000100:00080000:0.0:1787827761.375167:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827761.375172:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827761.375178:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827761.375191:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564188416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827761.375199:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827761.375202:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827761.375204:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827761.375207:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2701 00000800:00000100:3.0:1787827761.375219:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787827761.375220:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827761.375227:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827761.375233:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827761.375243:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564188544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827761.375251:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827761.375253:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827761.375255:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827761.375258:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2701 00000100:00080000:0.0:1787827761.375260:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827761.375263:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827761.375267:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827761.375276:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827761/real 1787827761] req@ffff976882aef0c0 x1874670564188160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827761.375277:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564188672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827761.375317:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564188160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827761.375332:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827761.375348:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827761.375371:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564188160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827761.375408:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564188288:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827761.375439:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827761.375471:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827761/real 1787827761] req@ffff976882aee940 x1874670564188288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827761.375486:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564188288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827761.375497:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827761.375501:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827761.375505:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564188288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827761.375512:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564188416:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827761.375536:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827761.375548:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827761/real 1787827761] req@ffff976882aee580 x1874670564188416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827761.375563:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564188416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827761.375573:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827761.375577:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827761.375581:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564188416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827761.375588:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564188544:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827761.375624:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827761.375635:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827761/real 1787827761] req@ffff976882aec3c0 x1874670564188544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827761.375658:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564188544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827761.375668:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827761.375671:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827761.375675:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564188544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827761.375682:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564188672:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827761.375701:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827761.375712:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827761/real 1787827761] req@ffff976882aeed00 x1874670564188672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827761.375726:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564188672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827761.375752:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827761.375756:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827761.375760:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564188672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827761.630875:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827761.630884:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827761.630896:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827762.398950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827762.654911:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827762.654941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827762.654970:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827763.422911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827763.678910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827763.678910:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827763.678935:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827764.446939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827764.702938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827764.702939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827764.702975:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827765.470893:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827765.726899:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827765.726903:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827765.726939:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827766.494935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827766.494961:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827766.494981:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827766.494987:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827766.494998:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2706 00000100:00080000:0.0:1787827766.495010:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827766.495018:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827766.495073:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564188800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827766.495101:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827766.495104:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827766.495106:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827766.495109:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2706 00000100:00080000:0.0:1787827766.495112:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827766.495116:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827766.495127:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827766.495152:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564188928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827766.495161:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564188800:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787827766.495186:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827766.495192:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827766.495196:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827766.495204:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2706 00000100:00080000:0.0:1787827766.495221:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827766.495225:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827766.495231:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827766.495243:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564189056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827766.495251:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827766.495254:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1787827766.495260:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787827766.495264:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827766.495270:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2706 00000100:00080000:0.0:1787827766.495274:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827766.495278:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827766.495284:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827766.495295:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564189184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827766.495307:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827766.495310:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827766.495312:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827766.495316:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2706 00000100:00080000:0.0:1787827766.495319:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827766.495324:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827766/real 1787827766] req@ffff976882aed680 x1874670564188800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827766.495328:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827766.495340:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827766.495360:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564189312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827766.495367:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564188800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827766.495393:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827766.495401:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827766.495406:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564188800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827766.495432:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564188928:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827766.495453:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827766.495464:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827766/real 1787827766] req@ffff976882aeed00 x1874670564188928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827766.495474:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564188928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827766.495481:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827766.495484:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827766.495487:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564188928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827766.495491:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564189056:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827766.495505:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827766.495512:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827766/real 1787827766] req@ffff976882aec3c0 x1874670564189056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827766.495521:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564189056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827766.495527:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827766.495530:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827766.495532:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564189056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827766.495536:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564189184:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827766.495550:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827766.495557:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827766/real 1787827766] req@ffff976882aee580 x1874670564189184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827766.495566:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564189184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827766.495572:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827766.495574:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827766.495576:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564189184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827766.495581:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564189312:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827766.495609:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827766.495629:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827766/real 1787827766] req@ffff976882aee940 x1874670564189312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827766.495640:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564189312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827766.495655:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827766.495674:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827766.495676:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564189312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827766.750931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827766.750931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827766.750963:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827767.518897:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827767.774958:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827767.774972:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827767.774991:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827768.542940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827768.798989:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827768.798989:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827768.799016:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827769.566989:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827769.822958:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827769.822974:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827769.823009:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827770.590924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827770.846914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827770.846938:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827770.846959:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00080000:0.0:1787827771.614926:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827771.614926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827771.614937:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827771.614941:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827771.614951:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2711 00000100:00080000:0.0:1787827771.614963:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827771.614971:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827771.615013:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564189440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827771.615045:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827771.615048:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827771.615049:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827771.615053:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2711 00000100:00080000:0.0:1787827771.615055:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827771.615059:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827771.615070:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827771.615078:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564189440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827771.615084:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564189568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827771.615094:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827771.615096:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827771.615097:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827771.615100:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2711 00000100:00080000:0.0:1787827771.615103:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827771.615105:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827771.615108:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827771.615117:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564189696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827771.615123:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827771.615125:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827771.615127:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827771.615130:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2711 00000100:00080000:0.0:1787827771.615134:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827771.615138:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827771.615145:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787827771.615155:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827771.615160:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564189824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827771.615173:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827771.615177:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827771.615181:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827771.615185:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2711 00000100:00080000:0.0:1787827771.615190:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827771.615193:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827771.615199:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827771.615203:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827771/real 1787827771] req@ffff976882aef0c0 x1874670564189440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827771.615211:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564189952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827771.615228:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564189440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827771.615242:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827771.615247:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827771.615252:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564189440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827771.615282:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564189568:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827771.615305:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827771.615317:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827771/real 1787827771] req@ffff976882aee940 x1874670564189568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827771.615326:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564189568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827771.615333:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827771.615335:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827771.615346:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564189568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827771.615351:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564189696:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827771.615364:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827771.615371:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827771/real 1787827771] req@ffff976882aee580 x1874670564189696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827771.615384:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564189696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827771.615393:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827771.615397:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827771.615401:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564189696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827771.615408:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564189824:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827771.615431:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827771.615440:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827771/real 1787827771] req@ffff976882aec3c0 x1874670564189824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827771.615448:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564189824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827771.615455:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827771.615457:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827771.615460:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564189824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827771.615464:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564189952:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827771.615476:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827771.615482:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827771/real 1787827771] req@ffff976882aeed00 x1874670564189952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827771.615489:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564189952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827771.615495:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827771.615496:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827771.615498:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564189952:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827771.870969:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827771.870977:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827771.871019:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827772.638910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827772.894973:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827772.894973:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827772.894996:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827773.662952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827773.918997:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827773.918997:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827773.919021:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827774.686900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827774.942937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827774.942937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827774.942974:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827775.710918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827775.966977:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827775.966978:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827775.967005:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827776.734919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827776.734930:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827776.734937:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827776.734941:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827776.734950:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2716 00000100:00080000:0.0:1787827776.734961:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827776.734969:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827776.735007:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564190080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827776.735033:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827776.735036:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827776.735038:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827776.735041:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2716 00000100:00080000:0.0:1787827776.735044:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827776.735049:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827776.735058:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827776.735068:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564190080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827776.735074:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564190208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827776.735083:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827776.735086:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827776.735087:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827776.735090:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2716 00000100:00080000:0.0:1787827776.735092:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827776.735094:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827776.735097:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827776.735107:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564190336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827776.735112:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827776.735114:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827776.735115:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827776.735118:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2716 00000100:00080000:0.0:1787827776.735121:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827776.735123:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827776.735127:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827776.735134:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564190464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827776.735140:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827776.735143:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827776.735144:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827776.735147:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2716 00000100:00080000:0.0:1787827776.735149:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827776.735152:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827776.735155:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827776.735165:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564190208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827776.735166:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564190592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827776.735208:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564190336:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827776.735227:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564190464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827776.735246:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564190592:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827776.990900:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827776.990906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827776.990932:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827777.758909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827778.014927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827778.014927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827778.014947:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827778.782906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827779.038892:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827779.038917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827779.038936:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827779.806900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:1.0:1787827779.807041:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:1.0:1787827779.807080:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787827779.807103:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827779.807147:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827779.807160:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827779.807169:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827779.807178:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:3.0:1787827779.807183:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827776/real 1787827779] req@ffff976882aeed00 x1874670564190208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827779.807212:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564190208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827779.807225:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827779.807230:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827779.807233:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564190208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827779.807239:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827776/real 1787827779] req@ffff976882aec3c0 x1874670564190336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827779.807248:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564190336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827779.807254:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827779.807257:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827779.807259:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564190336:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827779.807264:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827776/real 1787827779] req@ffff976882aee580 x1874670564190464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827779.807271:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564190464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827779.807275:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827779.807277:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827779.807279:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564190464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827779.807283:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827776/real 1787827779] req@ffff976882aed680 x1874670564190080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827779.807290:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564190080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827779.807295:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827779.807298:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827779.807301:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564190080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827779.807328:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827776/real 1787827779] req@ffff976882aee940 x1874670564190592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827779.807340:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564190592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827779.807348:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827779.807352:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827779.807354:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564190592:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827780.062919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827780.062923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827780.062951:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827780.830918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827781.086897:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827781.086899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827781.086917:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827781.854937:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827781.854941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827781.854946:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827781.854956:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827781.854968:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2721 00000100:00080000:0.0:1787827781.854980:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827781.854989:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827781.855024:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564190720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827781.855049:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827781.855052:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827781.855053:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827781.855056:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2721 00000100:00080000:0.0:1787827781.855060:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827781.855066:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827781.855078:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827781.855093:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564190720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827781.855097:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564190848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827781.855115:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827781.855120:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827781.855122:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827781.855127:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2721 00000100:00080000:0.0:1787827781.855131:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827781.855136:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827781.855142:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827781.855160:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564190976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827781.855170:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827781.855173:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827781.855174:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827781.855178:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2721 00000100:00080000:0.0:1787827781.855183:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827781.855187:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827781.855192:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827781.855208:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564191104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827781.855212:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787827781.855231:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827781.855234:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827781.855235:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827781.855241:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2721 00000100:00080000:0.0:1787827781.855252:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827781.855256:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827781.855260:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827781.855283:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564191232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827781.855293:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827781/real 1787827781] req@ffff976882aef0c0 x1874670564190720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827781.855325:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564190720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827781.855339:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827781.855345:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827781.855350:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564190720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827781.855390:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564190848:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827781.855422:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827781.855438:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827781/real 1787827781] req@ffff976882aee940 x1874670564190848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827781.855451:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564190848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827781.855460:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827781.855464:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827781.855469:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564190848:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827781.855476:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564190976:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827781.855499:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827781.855511:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827781/real 1787827781] req@ffff976882aed680 x1874670564190976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827781.855523:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564190976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827781.855531:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827781.855534:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827781.855538:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564190976:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827781.855544:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564191104:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827781.855587:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827781.855600:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827781/real 1787827781] req@ffff976882aee580 x1874670564191104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827781.855613:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564191104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827781.855622:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827781.855625:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827781.855628:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564191104:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827781.855637:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564191232:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827781.855660:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827781.855672:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827781/real 1787827781] req@ffff976882aec3c0 x1874670564191232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827781.855686:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564191232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827781.855693:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827781.855697:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827781.855700:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564191232:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827782.110901:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827782.110907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827782.110946:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827782.878982:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827783.134904:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827783.134910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827783.134926:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827783.902886:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827784.158944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827784.158964:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827784.158981:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827784.926982:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827785.182951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827785.182951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827785.182976:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827785.950912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827786.206918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827786.206918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827786.206936:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827786.974942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827786.974969:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827786.974976:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827786.974980:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827786.974988:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2726 00000100:00080000:1.0:1787827786.974998:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827786.975004:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827786.975050:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564191360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827786.975073:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827786.975076:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827786.975077:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827786.975080:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2726 00000100:00080000:1.0:1787827786.975083:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827786.975086:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827786.975094:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827786.975115:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564191488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827786.975122:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564191360:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827786.975129:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827786.975133:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827786.975134:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827786.975137:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2726 00000100:00080000:1.0:1787827786.975139:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827786.975142:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827786.975145:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827786.975156:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564191616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827786.975171:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827786.975174:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827786.975177:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827786.975182:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2726 00000100:00080000:1.0:1787827786.975187:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827786.975192:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827786.975198:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787827786.975204:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827786.975216:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564191744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827786.975229:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827786.975233:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827786.975235:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827786.975239:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2726 00000100:00080000:1.0:1787827786.975244:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827786.975250:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827786.975250:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827786/real 1787827786] req@ffff976889efc780 x1874670564191360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787827786.975258:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827786.975273:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564191872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827786.975284:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564191360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827786.975296:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827786.975300:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827786.975304:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564191360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827786.975325:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564191488:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827786.975350:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827786.975362:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827786/real 1787827786] req@ffff976889eff480 x1874670564191488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827786.975382:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564191488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827786.975389:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827786.975392:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827786.975394:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564191488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827786.975398:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564191616:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827786.975411:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827786.975418:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827786/real 1787827786] req@ffff976889efe580 x1874670564191616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827786.975429:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564191616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827786.975437:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827786.975441:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827786.975446:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564191616:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827786.975452:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564191744:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827786.975475:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827786.975489:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827786/real 1787827786] req@ffff976889efe1c0 x1874670564191744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827786.975508:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564191744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827786.975516:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827786.975520:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827786.975524:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564191744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827786.975531:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564191872:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827786.975556:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827786.975569:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827786/real 1787827786] req@ffff976889efcb40 x1874670564191872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827786.975581:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564191872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827786.975587:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827786.975589:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827786.975592:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564191872:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827787.230919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827787.230932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827787.230984:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827787.998900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827788.254988:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827788.254988:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827788.254998:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827789.022985:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827789.278913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827789.278948:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827789.278955:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787827790.046904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827790.302923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827790.302923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827790.302950:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827791.070917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827791.326942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827791.326942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827791.326969:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827792.094895:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827792.094958:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827792.094965:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827792.094970:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827792.094982:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2731 00000100:00080000:2.0:1787827792.094993:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827792.095000:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827792.095045:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564192000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827792.095071:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827792.095075:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827792.095077:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827792.095081:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2731 00000100:00080000:2.0:1787827792.095085:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827792.095098:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827792.095108:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827792.095123:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564192000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827792.095125:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564192128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827792.095137:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827792.095147:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827792.095150:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827792.095157:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2731 00000100:00080000:2.0:1787827792.095163:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827792.095170:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827792.095176:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787827792.095206:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827792.095213:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564192256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827792.095226:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827792.095229:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827792.095232:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827792.095239:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2731 00000100:00080000:2.0:1787827792.095246:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827792.095255:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827792.095256:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827792/real 1787827792] req@ffff976889cfad00 x1874670564192000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827792.095264:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827792.095285:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564192384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827792.095295:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564192000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827792.095302:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827792.095312:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827792.095316:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827792.095325:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2731 00000100:00080000:2.0:1787827792.095332:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827792.095340:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:0.0:1787827792.095343:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787827792.095346:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787827792.095354:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787827792.095361:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564192512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827792.095365:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564192000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827792.095399:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564192128:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827792.095430:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827792.095452:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827792/real 1787827792] req@ffff976889cf92c0 x1874670564192128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827792.095471:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564192128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827792.095494:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827792.095498:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827792.095503:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564192128:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827792.095509:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564192256:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827792.095535:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827792.095549:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827792/real 1787827792] req@ffff976889cfa1c0 x1874670564192256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827792.095560:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564192256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827792.095568:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827792.095571:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827792.095575:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564192256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827792.095581:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564192384:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827792.095597:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827792.095605:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827792/real 1787827792] req@ffff976889cfa580 x1874670564192384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827792.095614:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564192384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827792.095621:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827792.095623:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827792.095625:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564192384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827792.095629:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564192512:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827792.095642:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827792.095649:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827792/real 1787827792] req@ffff976889cfb480 x1874670564192512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827792.095657:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564192512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827792.095663:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827792.095666:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827792.095668:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564192512:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827792.350897:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827792.350897:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827792.350915:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827793.118933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827793.374892:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827793.374904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827793.374963:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827794.142879:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827794.398913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827794.398913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827794.398940:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827795.166942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827795.422993:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827795.422993:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827795.423032:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827796.190925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827796.446960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827796.446962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827796.447037:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827797.214965:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827797.214978:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827797.214985:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827797.214999:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2737 00000100:00100000:0.0:1787827797.215010:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827797.215013:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827797.215025:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827797.215076:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564192640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827797.215121:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827797.215126:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827797.215128:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827797.215134:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2737 00000100:00080000:3.0:1787827797.215138:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827797.215142:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827797.215154:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827797.215173:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564192768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827797.215209:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827797.215215:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564192640:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827797.215216:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827797.215220:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827797.215230:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2737 00000100:00080000:3.0:1787827797.215236:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827797.215240:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827797.215246:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827797.215283:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564192896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827797.215296:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827797.215301:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827797.215303:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827797.215308:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2737 00000100:00080000:3.0:1787827797.215313:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827797.215318:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827797.215324:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1787827797.215334:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827797.215343:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564193024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827797.215360:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827797.215364:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827797.215367:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827797.215371:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2737 00000100:00080000:3.0:1787827797.215377:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827797.215383:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827797.215389:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827797.215398:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827797/real 1787827797] req@ffff97688cb5de00 x1874670564192640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827797.215407:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564193152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827797.215463:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564192640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827797.215486:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827797.215495:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827797.215514:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564192640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827797.215564:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564192768:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827797.215605:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827797.215624:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827797/real 1787827797] req@ffff97688cb5f0c0 x1874670564192768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827797.215641:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564192768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827797.215679:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827797.215693:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827797.215698:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564192768:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827797.215706:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564192896:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827797.215736:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827797.215752:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827797/real 1787827797] req@ffff97688cb5d680 x1874670564192896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827797.215767:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564192896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827797.215778:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827797.215782:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827797.215789:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564192896:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827797.215801:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564193024:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827797.215899:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827797.215919:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827797/real 1787827797] req@ffff97688cb5c000 x1874670564193024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827797.215935:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564193024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827797.215946:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827797.215950:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827797.215954:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564193024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827797.215962:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564193152:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827797.215986:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827797.215997:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827797/real 1787827797] req@ffff97688cb5cb40 x1874670564193152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827797.216011:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564193152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827797.216022:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827797.216025:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827797.216029:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564193152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827797.470950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827797.470999:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827797.471004:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827798.238957:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827798.495000:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827798.495006:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827798.495135:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827799.262937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827799.518947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827799.518952:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827799.518976:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827800.286930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827800.542924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827800.542924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827800.543020:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827801.311006:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827801.567024:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827801.567042:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827801.567090:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827802.334923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827802.334940:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827802.334950:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827802.334957:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827802.334971:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2742 00000100:00080000:3.0:1787827802.334986:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827802.334996:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827802.335052:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564193280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827802.335091:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827802.335096:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827802.335098:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827802.335103:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2742 00000100:00080000:3.0:1787827802.335107:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827802.335113:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827802.335127:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827802.335138:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564193280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827802.335146:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564193408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827802.335161:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827802.335165:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827802.335167:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827802.335172:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2742 00000100:00080000:3.0:1787827802.335176:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827802.335180:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827802.335186:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827802.335209:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564193536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827802.335219:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827802.335223:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827802.335225:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827802.335230:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2742 00000100:00080000:3.0:1787827802.335234:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827802.335238:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827802.335244:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1787827802.335250:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827802.335260:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564193664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827802.335272:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827802.335276:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827802.335279:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827802.335283:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2742 00000100:00080000:3.0:1787827802.335289:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827802.335293:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827802.335299:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827802.335305:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827802/real 1787827802] req@ffff97688cb5f840 x1874670564193280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827802.335317:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564193792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827802.335326:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564193280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827802.335342:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827802.335347:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827802.335352:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564193280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827802.335393:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564193408:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827802.335422:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827802.335434:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827802/real 1787827802] req@ffff97688cb5c3c0 x1874670564193408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827802.335444:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564193408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827802.335450:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827802.335453:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827802.335455:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564193408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827802.335462:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564193536:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827802.335481:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827802.335493:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827802/real 1787827802] req@ffff97688cb5e1c0 x1874670564193536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827802.335508:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564193536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827802.335517:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827802.335519:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827802.335522:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564193536:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827802.335526:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564193664:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827802.335543:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827802.335550:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827802/real 1787827802] req@ffff97688cb5c780 x1874670564193664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827802.335558:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564193664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827802.335564:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827802.335567:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827802.335569:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564193664:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827802.335572:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564193792:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827802.335583:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827802.335589:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827802/real 1787827802] req@ffff97688cb5cf00 x1874670564193792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827802.335596:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564193792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827802.335602:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827802.335604:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827802.335613:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564193792:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827802.590968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827802.591036:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827802.591062:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827803.358909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827803.614976:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827803.615003:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827803.615040:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827804.382958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827804.638935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827804.639008:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827804.639012:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827805.406937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827805.662906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827805.662946:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827805.662985:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787827806.431057:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827806.686954:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827806.686966:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827806.687105:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827807.454920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827807.455011:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827807.455023:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827807.455030:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827807.455046:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2747 00000100:00080000:1.0:1787827807.455062:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827807.455075:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827807.455135:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564193920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827807.455179:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827807.455184:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827807.455186:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827807.455193:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2747 00000100:00080000:1.0:1787827807.455198:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827807.455203:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827807.455216:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827807.455234:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564194048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827807.455250:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827807.455255:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827807.455258:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827807.455267:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2747 00000100:00100000:0.0:1787827807.455271:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564193920:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827807.455271:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827807.455281:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827807.455295:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827807.455310:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564194176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827807.455322:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827807.455326:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827807.455328:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827807.455334:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2747 00000100:00080000:1.0:1787827807.455339:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827807.455344:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827807.455351:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827807.455364:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564194304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827807.455375:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827807.455379:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827807.455385:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787827807.455388:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787827807.455394:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2747 00000100:00080000:1.0:1787827807.455402:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827807.455406:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827807.455415:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827807.455435:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564194432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827807.455493:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827807/real 1787827807] req@ffff976889eff0c0 x1874670564193920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827807.455521:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564193920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827807.455535:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827807.455541:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827807.455546:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564193920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827807.455606:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564194048:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827807.455652:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827807.455672:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827807/real 1787827807] req@ffff976889efed00 x1874670564194048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827807.455689:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564194048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827807.455701:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827807.455705:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827807.455710:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564194048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827807.455718:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564194176:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827807.455740:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827807.455752:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827807/real 1787827807] req@ffff976889efc3c0 x1874670564194176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827807.455767:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564194176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827807.455787:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827807.455791:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827807.455795:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564194176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827807.455802:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564194304:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827807.455821:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827807.455875:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827807/real 1787827807] req@ffff976889efe940 x1874670564194304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827807.455890:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564194304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827807.455901:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827807.455905:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827807.455909:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564194304:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827807.455917:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564194432:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827807.455946:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827807.455967:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827807/real 1787827807] req@ffff976889effc00 x1874670564194432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827807.455991:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564194432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827807.456008:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827807.456015:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827807.456023:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564194432:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827807.710971:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827807.711006:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827807.711033:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827808.478949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827808.734971:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827808.735032:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827808.735049:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827809.502899:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827809.758958:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827809.758993:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827809.759012:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827810.526896:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827810.782988:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827810.782994:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827810.783085:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827811.550876:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827811.806922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827811.806926:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827811.806993:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00080000:3.0:1787827812.574904:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827812.574912:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827812.574917:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827812.574926:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2752 00000100:00100000:0.0:1787827812.574930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827812.574935:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827812.574943:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827812.574975:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564194560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827812.575002:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827812.575005:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827812.575007:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827812.575010:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2752 00000100:00080000:3.0:1787827812.575013:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827812.575017:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827812.575025:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827812.575033:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564194560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827812.575039:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564194688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827812.575048:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827812.575051:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827812.575053:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827812.575056:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2752 00000100:00080000:3.0:1787827812.575058:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827812.575060:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827812.575064:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827812.575072:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564194816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827812.575078:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827812.575081:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827812.575082:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827812.575085:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2752 00000100:00080000:3.0:1787827812.575088:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827812.575090:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:0.0:1787827812.575095:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:3.0:1787827812.575096:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827812.575107:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564194944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827812.575114:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827812.575116:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827812.575118:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827812.575120:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2752 00000100:00080000:3.0:1787827812.575123:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827812.575127:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827812.575130:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827812.575132:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827812/real 1787827812] req@ffff97688cb5d2c0 x1874670564194560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827812.575141:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564195072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827812.575155:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564194560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827812.575166:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827812.575170:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827812.575173:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564194560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827812.575192:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564194688:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827812.575208:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827812.575217:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827812/real 1787827812] req@ffff97688cb5cf00 x1874670564194688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827812.575226:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564194688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827812.575232:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827812.575235:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827812.575238:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564194688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827812.575242:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564194816:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827812.575253:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827812.575259:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827812/real 1787827812] req@ffff97688cb5c780 x1874670564194816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827812.575266:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564194816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827812.575272:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827812.575274:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827812.575276:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564194816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827812.575279:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564194944:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827812.575290:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827812.575296:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827812/real 1787827812] req@ffff97688cb5e1c0 x1874670564194944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827812.575303:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564194944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827812.575307:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827812.575310:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827812.575312:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564194944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827812.575316:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564195072:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827812.575327:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827812.575336:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827812/real 1787827812] req@ffff97688cb5c3c0 x1874670564195072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827812.575349:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564195072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827812.575358:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827812.575362:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827812.575366:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564195072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827812.830928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827812.830969:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827812.831016:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827813.598983:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827813.854972:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827813.855032:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827813.855051:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827814.622935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827814.878980:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827814.879078:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827814.879099:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827815.646973:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827815.902883:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827815.902935:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827815.902959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827816.670966:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827816.926930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827816.926966:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827816.927014:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00080000:3.0:1787827817.694955:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827817.694969:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1787827817.694970:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827817.694975:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827817.694991:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2757 00000100:00080000:3.0:1787827817.695005:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827817.695015:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827817.695077:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564195200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827817.695112:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827817.695117:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827817.695120:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827817.695126:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2757 00000100:00080000:3.0:1787827817.695131:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827817.695137:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827817.695151:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827817.695170:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564195200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827817.695173:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564195328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827817.695195:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827817.695200:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827817.695202:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827817.695208:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2757 00000100:00080000:3.0:1787827817.695212:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827817.695217:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827817.695224:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827817.695238:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564195456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827817.695250:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827817.695255:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827817.695258:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787827817.695268:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827817.695274:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2757 00000100:00080000:3.0:1787827817.695284:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827817.695290:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827817.695297:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827817.695312:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564195584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827817.695325:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827817.695329:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827817.695333:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827817.695345:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827817/real 1787827817] req@ffff97688cb5f840 x1874670564195200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827817.695345:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2757 00000100:00080000:3.0:1787827817.695358:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827817.695373:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827817.695388:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827817.695401:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564195200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827817.695419:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564195712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827817.695425:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827817.695440:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827817.695453:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564195200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827817.695497:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564195328:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827817.695558:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827817.695579:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827817/real 1787827817] req@ffff97688cb5cb40 x1874670564195328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827817.695595:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564195328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827817.695607:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827817.695611:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827817.695616:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564195328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827817.695623:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564195456:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827817.695644:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827817.695656:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827817/real 1787827817] req@ffff97688cb5c000 x1874670564195456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827817.695674:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564195456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827817.695690:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827817.695696:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827817.695703:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564195456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827817.695714:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564195584:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827817.695755:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827817.695779:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827817/real 1787827817] req@ffff97688cb5d680 x1874670564195584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827817.695796:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564195584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827817.695806:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827817.695810:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827817.695814:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564195584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827817.695853:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564195712:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827817.695885:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827817.695900:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827817/real 1787827817] req@ffff97688cb5f0c0 x1874670564195712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827817.695914:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564195712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827817.695925:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827817.695929:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827817.695933:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564195712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827817.950918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827817.950991:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827817.951024:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827818.718978:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827818.974998:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827818.974998:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827818.975104:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827819.742918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827819.998904:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827819.998904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827819.999077:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827820.766996:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827821.022982:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827821.023045:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827821.023112:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827821.790927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827822.047047:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827822.047047:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827822.047101:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827822.814989:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827822.815003:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827822.815010:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827822.815023:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2762 00000100:00080000:1.0:1787827822.815039:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827822.815050:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827822.815067:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827822.815118:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564195840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827822.815152:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827822.815157:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827822.815159:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827822.815165:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2762 00000100:00080000:1.0:1787827822.815170:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827822.815176:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827822.815189:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827822.815213:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564195840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827822.815213:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564195968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827822.815242:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827822.815246:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827822.815248:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827822.815253:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2762 00000100:00080000:1.0:1787827822.815256:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827822.815260:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827822.815266:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827822.815283:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564196096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827822.815296:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827822.815301:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827822.815304:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827822.815310:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2762 00000100:00080000:1.0:1787827822.815326:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827822.815331:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827822.815339:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1787827822.815350:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827822.815365:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564196224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827822.815410:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827822.815417:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827822.815421:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827822.815431:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827822/real 1787827822] req@ffff976889effc00 x1874670564195840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827822.815437:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2762 00000100:00080000:1.0:1787827822.815452:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827822.815469:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827822.815507:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827822.815513:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564195840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827822.815534:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827822.815544:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827822.815550:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564195840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827822.815557:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564196352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827822.815588:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564195968:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827822.815633:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827822.815654:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827822/real 1787827822] req@ffff976889efe940 x1874670564195968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827822.815672:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564195968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827822.815686:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827822.815692:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827822.815696:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564195968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827822.815704:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564196096:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827822.815741:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827822.815761:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827822/real 1787827822] req@ffff976889efc3c0 x1874670564196096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827822.815777:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564196096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827822.815788:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827822.815792:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827822.815796:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564196096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827822.815803:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564196224:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827822.815852:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827822.815878:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827822/real 1787827822] req@ffff976889efed00 x1874670564196224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827822.815903:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564196224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827822.815920:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827822.815927:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827822.815934:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564196224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827822.815969:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564196352:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827822.816011:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827822.816038:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827822/real 1787827822] req@ffff976889eff0c0 x1874670564196352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827822.816067:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564196352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827822.816078:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827822.816082:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827822.816087:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564196352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827823.070942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827823.071024:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827823.071096:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827823.838930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827824.094985:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827824.095061:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827824.095130:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827824.862992:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827825.118919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827825.118948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827825.119044:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827825.886916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827826.142904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827826.142931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827826.142952:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827826.910930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827827.166887:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827827.166938:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827827.166968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827827.934942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827827.934954:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827827.934974:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827827.934981:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827827.934993:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2767 00000100:00080000:1.0:1787827827.935005:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827827.935015:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827827.935077:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564196480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827827.935113:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827827.935119:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827827.935123:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827827.935132:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2767 00000100:00080000:1.0:1787827827.935140:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827827.935147:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827827.935169:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564196480:192.168.203.139@tcp:250:kworker.0 00000008:00100000:1.0:1787827827.935172:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827827.935201:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564196608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827827.935222:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827827.935228:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827827.935231:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827827.935239:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2767 00000100:00080000:1.0:1787827827.935259:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827827.935265:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827827.935282:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787827827.935286:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827827.935301:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564196736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827827.935314:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827827.935319:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827827.935322:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827827.935331:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2767 00000100:00080000:1.0:1787827827.935337:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827827.935347:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827827.935348:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827827/real 1787827827] req@ffff976889efcb40 x1874670564196480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787827827.935356:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827827.935378:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564196864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827827.935383:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564196480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827827.935415:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827827.935420:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827827.935420:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827827.935423:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827827.935429:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827827.935454:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564196480:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827827.935456:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2767 00000100:00080000:1.0:1787827827.935469:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827827.935477:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827827.935483:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827827.935495:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564196608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827827.935500:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564196992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827827.935559:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827827.935587:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827827/real 1787827827] req@ffff976889efe1c0 x1874670564196608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827827.935604:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564196608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827827.935613:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827827.935617:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827827.935620:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564196608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827827.935628:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564196736:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827827.935661:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827827.935687:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827827/real 1787827827] req@ffff976889efe580 x1874670564196736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827827.935711:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564196736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827827.935731:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827827.935738:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827827.935745:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564196736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827827.935753:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564196864:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827827.935793:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827827.935808:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827827/real 1787827827] req@ffff976889eff480 x1874670564196864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827827.935850:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564196864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827827.935867:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827827.935871:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827827.935876:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564196864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827827.935905:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564196992:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827827.935926:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827827.935938:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827827/real 1787827827] req@ffff976889efc780 x1874670564196992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827827.935950:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564196992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827827.935963:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827827.935968:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827827.935972:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564196992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827828.190887:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827828.190916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827828.190946:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827828.958913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827829.214927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827829.214927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827829.215001:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827829.982937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827830.238886:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827830.238895:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827830.238916:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827831.006902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827831.262886:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827831.262890:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827831.262957:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827832.030944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827832.286926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827832.286926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827832.286953:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827833.054931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827833.054944:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827833.054950:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827833.054954:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827833.054963:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2772 00000100:00080000:2.0:1787827833.054974:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827833.054982:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827833.055023:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564197120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827833.055049:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827833.055052:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827833.055054:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827833.055057:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2772 00000100:00080000:2.0:1787827833.055060:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827833.055064:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827833.055073:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827833.055086:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564197248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827833.055094:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564197120:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787827833.055097:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827833.055102:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827833.055104:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827833.055108:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2772 00000100:00080000:2.0:1787827833.055110:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827833.055113:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827833.055117:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827833.055125:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564197376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827833.055133:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827833.055136:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827833.055138:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827833.055140:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2772 00000100:00080000:2.0:1787827833.055143:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827833.055146:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827833.055149:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1787827833.055157:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827833.055160:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564197504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827833.055168:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827833.055171:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827833.055173:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827833.055177:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2772 00000100:00080000:2.0:1787827833.055180:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827833.055183:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827833.055186:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827833.055200:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827833/real 1787827833] req@ffff976889cf9e00 x1874670564197120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827833.055201:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564197632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827833.055253:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564197120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827833.055286:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827833.055291:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827833.055295:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564197120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827833.055320:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564197248:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827833.055346:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827833.055361:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827833/real 1787827833] req@ffff976889cfa940 x1874670564197248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827833.055373:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564197248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827833.055382:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827833.055385:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827833.055389:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564197248:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827833.055431:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564197376:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827833.055449:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827833.055459:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827833/real 1787827833] req@ffff976889cfbc00 x1874670564197376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827833.055470:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564197376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827833.055478:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827833.055482:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827833.055492:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564197376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827833.055498:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564197504:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827833.055514:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827833.055526:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827833/real 1787827833] req@ffff976889cf8000 x1874670564197504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827833.055536:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564197504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827833.055544:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827833.055548:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827833.055553:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564197504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827833.055560:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564197632:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827833.055603:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827833.055620:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827833/real 1787827833] req@ffff976889cfb0c0 x1874670564197632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827833.055637:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564197632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827833.055645:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827833.055648:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827833.055651:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564197632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827833.310917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827833.310936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827833.311031:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827834.078958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827834.334915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827834.334939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827834.334968:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827835.102943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827835.358879:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827835.358902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827835.358951:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827836.126921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827836.382943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827836.382943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827836.382972:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827837.150965:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827837.406945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827837.407001:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827837.407075:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00080000:2.0:1787827838.174961:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827838.174968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827838.174972:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827838.174981:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827838.174995:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2778 00000100:00080000:2.0:1787827838.175009:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827838.175017:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827838.175055:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564197760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827838.175081:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827838.175083:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827838.175085:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827838.175088:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2778 00000100:00080000:2.0:1787827838.175091:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827838.175093:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827838.175103:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827838.175113:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564197888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827838.175121:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827838.175126:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787827838.175127:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564197760:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787827838.175128:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827838.175135:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2778 00000100:00080000:2.0:1787827838.175138:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827838.175140:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827838.175143:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827838.175151:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564198016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827838.175157:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827838.175160:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827838.175161:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827838.175164:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2778 00000100:00080000:2.0:1787827838.175167:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827838.175169:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827838.175171:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827838.175180:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564198144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827838.175186:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787827838.175192:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827838.175195:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827838.175198:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827838.175202:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2778 00000100:00080000:2.0:1787827838.175205:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827838.175208:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827838.175211:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827838.175218:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564198272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827838.175229:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827838/real 1787827838] req@ffff976889cf8f00 x1874670564197760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827838.175245:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564197760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827838.175254:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827838.175257:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827838.175260:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564197760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827838.175277:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564197888:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827838.175292:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827838.175302:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827838/real 1787827838] req@ffff976889cfb0c0 x1874670564197888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827838.175310:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564197888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827838.175316:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827838.175318:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827838.175320:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564197888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827838.175324:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564198016:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827838.175334:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827838.175340:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827838/real 1787827838] req@ffff976889cf8000 x1874670564198016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827838.175352:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564198016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827838.175358:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827838.175359:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827838.175361:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564198016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827838.175381:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564198144:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827838.175399:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827838.175425:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827838/real 1787827838] req@ffff976889cfbc00 x1874670564198144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827838.175435:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564198144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827838.175442:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827838.175445:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827838.175447:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564198144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827838.175452:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564198272:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827838.175465:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827838.175472:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827838/real 1787827838] req@ffff976889cfa940 x1874670564198272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827838.175481:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564198272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827838.175493:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827838.175495:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827838.175498:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564198272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827838.430967:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827838.431025:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827838.431029:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827839.198901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827839.454927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827839.454927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827839.454972:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827840.222908:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827840.478921:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827840.478921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827840.478962:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827841.246884:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827841.502915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827841.502916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827841.502975:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827842.270910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827842.526926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827842.526935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827842.526961:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827843.294891:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827843.294893:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827843.294901:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827843.294905:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827843.294912:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2783 00000100:00080000:2.0:1787827843.294922:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827843.294928:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827843.294961:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564198400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827843.294990:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827843.294994:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827843.294996:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827843.295001:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2783 00000100:00080000:2.0:1787827843.295005:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827843.295010:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827843.295020:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827843.295031:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564198528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827843.295039:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827843.295042:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827843.295044:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827843.295048:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2783 00000100:00080000:2.0:1787827843.295052:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827843.295056:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827843.295061:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827843.295064:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564198400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827843.295076:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564198656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827843.295089:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827843.295092:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827843.295094:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827843.295098:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2783 00000100:00080000:2.0:1787827843.295101:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827843.295104:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827843.295109:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827843.295119:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564198784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827843.295137:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827843.295141:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827843.295143:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827843.295148:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2783 00000100:00080000:2.0:1787827843.295152:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827843.295158:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564198528:192.168.203.139@tcp:38:kworker.0 00000100:00080000:2.0:1787827843.295160:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827843.295171:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827843.295183:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564198912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827843.295197:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564198656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827843.295217:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564198784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827843.295232:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564198912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827843.550908:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827843.550911:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827843.550962:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827844.318918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827844.574907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827844.574929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827844.574949:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827845.342877:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827845.598899:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827845.598912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827845.598991:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827846.366913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:1.0:1787827846.367083:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:1.0:1787827846.367123:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787827846.367140:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827846.367179:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827846.367188:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827846.367196:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787827846.367205:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:0.0:1787827846.367263:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827843/real 1787827846] req@ffff976889cfa580 x1874670564198656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827846.367285:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564198656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827846.367297:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827846.367303:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827846.367307:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564198656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827846.367329:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827843/real 1787827846] req@ffff976889cfa1c0 x1874670564198784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827846.367343:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564198784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827846.367352:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827846.367355:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827846.367359:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564198784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827846.367365:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827843/real 1787827846] req@ffff976889cf92c0 x1874670564198912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827846.367378:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564198912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827846.367385:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827846.367387:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827846.367390:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564198912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827846.367394:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827843/real 1787827846] req@ffff976889cf9e00 x1874670564198400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827846.367402:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564198400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827846.367408:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827846.367411:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827846.367414:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564198400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827846.367427:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827843/real 1787827846] req@ffff976889cfb480 x1874670564198528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827846.367440:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564198528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827846.367448:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827846.367457:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827846.367460:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564198528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787827846.622917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827846.622920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827846.622939:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827847.390943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827847.646898:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827847.646898:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827847.646980:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827848.415000:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827848.415000:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827848.415033:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827848.415041:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827848.415056:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2788 00000100:00080000:2.0:1787827848.415072:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827848.415085:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827848.415140:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564199040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827848.415185:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827848.415190:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827848.415193:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827848.415215:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2788 00000100:00080000:2.0:1787827848.415221:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827848.415226:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827848.415240:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827848.415259:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564199168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827848.415273:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827848.415278:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827848.415280:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827848.415288:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564199040:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787827848.415315:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2788 00000100:00080000:2.0:1787827848.415330:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827848.415350:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827848.415361:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827848.415388:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564199296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827848.415408:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827848.415415:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827848.415419:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827848.415427:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2788 00000100:00080000:2.0:1787827848.415433:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827848.415440:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:0.0:1787827848.415451:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:2.0:1787827848.415453:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827848.415481:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564199424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827848.415521:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827848.415526:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827848.415529:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827848.415535:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2788 00000100:00080000:2.0:1787827848.415542:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827848.415551:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827848.415561:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827848.415585:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827848/real 1787827848] req@ffff976889cfad00 x1874670564199040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827848.415607:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564199552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827848.415630:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564199040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827848.415669:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827848.415678:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827848.415687:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564199040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827848.415732:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564199168:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827848.415781:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827848.415810:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827848/real 1787827848] req@ffff976889cfb480 x1874670564199168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827848.415894:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564199168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827848.415924:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827848.415932:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827848.415939:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564199168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827848.415988:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564199296:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827848.416040:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827848.416064:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827848/real 1787827848] req@ffff976889cf9e00 x1874670564199296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827848.416088:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564199296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827848.416104:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827848.416110:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827848.416117:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564199296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827848.416129:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564199424:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827848.416172:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827848.416195:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827848/real 1787827848] req@ffff976889cf92c0 x1874670564199424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827848.416219:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564199424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827848.416235:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827848.416242:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827848.416250:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564199424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827848.416262:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564199552:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827848.416332:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827848.416355:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827848/real 1787827848] req@ffff976889cfa1c0 x1874670564199552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827848.416377:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564199552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827848.416410:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827848.416417:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827848.416424:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564199552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827848.670964:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827848.670964:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827848.670998:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827849.439002:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827849.694972:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827849.695000:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827849.695064:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827850.462881:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827850.718893:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827850.718969:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827850.719025:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827851.486952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827851.742958:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827851.742958:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827851.743040:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827852.510900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827852.766903:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827852.766961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827852.766981:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827853.534965:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827853.535004:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827853.535018:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827853.535042:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827853.535061:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2793 00000100:00080000:0.0:1787827853.535080:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827853.535094:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827853.535179:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564199680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827853.535230:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827853.535236:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827853.535238:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827853.535246:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2793 00000100:00080000:0.0:1787827853.535254:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827853.535285:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827853.535312:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564199680:192.168.203.139@tcp:250:kworker.0 00000008:00100000:0.0:1787827853.535314:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827853.535347:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564199808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827853.535365:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827853.535370:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827853.535373:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827853.535379:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2793 00000100:00080000:0.0:1787827853.535388:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827853.535393:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827853.535399:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827853.535444:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564199936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827853.535456:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827853.535459:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827853.535462:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787827853.535469:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787827853.535475:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2793 00000100:00080000:0.0:1787827853.535482:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827853.535487:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827853.535495:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827853.535509:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564200064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827853.535522:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827853.535526:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827853.535529:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827853.535537:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2793 00000100:00100000:3.0:1787827853.535538:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827853/real 1787827853] req@ffff976882aeed00 x1874670564199680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827853.535542:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827853.535557:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827853.535569:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827853.535574:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564199680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827853.535591:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787827853.535599:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564200192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827853.535600:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827853.535629:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564199680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827853.535670:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564199808:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827853.535707:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827853.535722:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827853/real 1787827853] req@ffff976882aec3c0 x1874670564199808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827853.535748:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564199808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827853.535758:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827853.535761:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827853.535765:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564199808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827853.535772:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564199936:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827853.535790:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827853.535801:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827853/real 1787827853] req@ffff976882aee580 x1874670564199936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827853.535848:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564199936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827853.535859:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827853.535862:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827853.535866:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564199936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827853.535872:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564200064:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827853.535894:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827853.535906:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827853/real 1787827853] req@ffff976882aed680 x1874670564200064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827853.535920:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564200064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827853.535929:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827853.535932:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827853.535936:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564200064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827853.535944:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564200192:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827853.535974:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827853.535993:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827853/real 1787827853] req@ffff976882aee940 x1874670564200192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827853.536015:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564200192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827853.536041:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827853.536045:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827853.536049:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564200192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827853.790973:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827853.791066:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827853.791109:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787827854.558919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827854.814944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827854.814951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827854.814979:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827855.582923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827855.838942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827855.838948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827855.839020:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827856.606913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827856.862930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827856.862931:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827856.862949:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827857.630899:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827857.887027:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827857.887074:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827857.887158:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827858.654910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827858.655076:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827858.655092:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827858.655100:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827858.655116:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2798 00000100:00080000:1.0:1787827858.655135:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827858.655149:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827858.655264:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564200320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827858.655304:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827858.655325:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827858.655328:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827858.655335:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2798 00000100:00080000:1.0:1787827858.655341:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827858.655346:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827858.655357:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564200320:192.168.203.139@tcp:250:kworker.0 00000008:00100000:1.0:1787827858.655361:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827858.655383:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564200448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827858.655398:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827858.655404:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827858.655407:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827858.655413:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2798 00000100:00080000:1.0:1787827858.655419:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827858.655424:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:3.0:1787827858.655432:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787827858.655437:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827858.655454:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564200576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827858.655478:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827858.655481:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827858.655484:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827858.655491:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2798 00000100:00080000:1.0:1787827858.655495:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827858.655502:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827858.655503:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827858/real 1787827858] req@ffff976889efc780 x1874670564200320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787827858.655512:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827858.655534:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564200704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827858.655543:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564200320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827858.655581:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827858.655590:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827858.655591:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827858.655598:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827858.655602:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:1.0:1787827858.655610:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2798 00000100:00100000:3.0:1787827858.655617:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564200320:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827858.655630:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827858.655645:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827858.655656:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827858.655687:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564200448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827858.655701:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564200832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827858.655719:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827858.655732:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827858/real 1787827858] req@ffff976889eff480 x1874670564200448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827858.655764:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564200448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827858.655771:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827858.655775:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827858.655777:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564200448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827858.655784:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564200576:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827858.655818:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827858.655825:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827858/real 1787827858] req@ffff976889efe580 x1874670564200576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827858.655834:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564200576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827858.655840:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827858.655842:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827858.655844:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564200576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827858.655848:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564200704:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827858.655900:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827858.655907:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827858/real 1787827858] req@ffff976889efe1c0 x1874670564200704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827858.655915:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564200704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827858.655924:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827858.655926:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827858.655928:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564200704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827858.655944:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564200832:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827858.655955:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827858.655961:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827858/real 1787827858] req@ffff976889efcb40 x1874670564200832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827858.655968:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564200832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827858.655973:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827858.655975:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827858.655977:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564200832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827858.910938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827858.910979:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827858.911002:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827859.678922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827859.934960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827859.935008:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827859.935028:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827860.702917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827860.958957:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827860.958987:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827860.959016:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787827861.726908:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827861.982930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827861.982942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827861.982953:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827862.750877:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827863.006894:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827863.006895:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827863.006951:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827863.775021:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827863.775025:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827863.775051:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827863.775062:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827863.775094:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2803 00000100:00080000:1.0:1787827863.775116:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827863.775133:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827863.775232:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564200960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827863.775304:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827863.775314:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827863.775331:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827863.775342:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2803 00000100:00080000:1.0:1787827863.775358:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827863.775363:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564200960:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827863.775376:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827863.775402:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827863.775459:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564201088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827863.775484:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827863.775491:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827863.775495:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827863.775504:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2803 00000100:00080000:1.0:1787827863.775511:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827863.775519:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827863.775541:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827863.775562:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564201216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827863.775576:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827863.775581:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827863.775597:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827863.775603:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2803 00000100:00080000:1.0:1787827863.775618:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827863.775623:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827863.775630:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827863.775644:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564201344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827863.775658:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827863.775662:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827863.775664:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827863.775668:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2803 00000100:00080000:1.0:1787827863.775672:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827863.775684:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827863.775689:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827863.775702:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564201472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827863.775979:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827863.776058:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827863/real 1787827863] req@ffff976889eff0c0 x1874670564200960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827863.776085:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564200960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827863.776109:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827863.776115:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827863.776120:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564200960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827863.776156:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564201088:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827863.776219:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827863.776251:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827863/real 1787827863] req@ffff976889efed00 x1874670564201088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827863.776268:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564201088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827863.776280:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827863.776284:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827863.776289:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564201088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827863.776297:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564201216:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827863.776333:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827863.776346:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827863/real 1787827863] req@ffff976889efc3c0 x1874670564201216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827863.776370:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564201216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827863.776380:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827863.776384:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827863.776389:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564201216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827863.776401:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564201344:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827863.776439:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827863.776464:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827863/real 1787827863] req@ffff976889efe940 x1874670564201344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827863.776490:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564201344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827863.776517:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827863.776522:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827863.776526:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564201344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827863.776535:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564201472:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827863.776565:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827863.776586:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827863/real 1787827863] req@ffff976889effc00 x1874670564201472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827863.776600:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564201472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827863.776619:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827863.776623:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827863.776627:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564201472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827864.030922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827864.030928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827864.030961:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827864.798914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827865.054908:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827865.054944:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827865.054980:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787827865.822906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827866.078900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827866.078909:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827866.078946:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827866.846939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827867.102914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827867.102915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827867.102950:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827867.870889:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827868.126911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827868.126911:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827868.126937:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827868.895003:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827868.895028:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827868.895034:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827868.895044:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827868.895049:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2808 00000100:00080000:1.0:1787827868.895067:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827868.895076:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827868.895168:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564201600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827868.895203:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827868.895224:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827868.895226:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827868.895231:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2808 00000100:00080000:1.0:1787827868.895235:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827868.895240:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827868.895254:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827868.895270:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564201728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827868.895283:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827868.895288:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827868.895290:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827868.895294:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2808 00000100:00080000:1.0:1787827868.895299:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827868.895310:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564201600:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827868.895311:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827868.895339:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827868.895353:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564201856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827868.895367:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827868.895372:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827868.895374:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827868.895378:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2808 00000100:00080000:1.0:1787827868.895385:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827868.895388:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827868.895406:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827868.895420:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564201984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827868.895431:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827868.895438:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1787827868.895442:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787827868.895443:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827868.895454:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2808 00000100:00080000:1.0:1787827868.895458:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827868.895461:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827868.895466:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827868.895480:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564202112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827868.895548:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827868/real 1787827868] req@ffff976889effc00 x1874670564201600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827868.895600:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564201600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827868.895622:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827868.895631:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827868.895639:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564201600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827868.895655:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564201728:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827868.895703:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827868.895730:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827868/real 1787827868] req@ffff976889efe940 x1874670564201728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827868.895756:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564201728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827868.895785:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827868.895789:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827868.895793:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564201728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827868.895823:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564201856:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827868.895923:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827868.895949:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827868/real 1787827868] req@ffff976889efc3c0 x1874670564201856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827868.895986:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564201856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827868.895996:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827868.896000:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827868.896004:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564201856:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827868.896012:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564201984:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827868.896033:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827868.896045:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827868/real 1787827868] req@ffff976889efed00 x1874670564201984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827868.896082:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564201984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827868.896098:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827868.896105:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827868.896112:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564201984:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827868.896124:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564202112:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827868.896201:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827868.896213:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827868/real 1787827868] req@ffff976889eff0c0 x1874670564202112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827868.896227:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564202112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827868.896236:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827868.896239:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827868.896243:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564202112:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827869.150935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827869.150942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827869.150964:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827869.918962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827870.174937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827870.174943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827870.174988:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827870.942992:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827871.198990:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827871.199003:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827871.199058:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827871.966937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827872.222944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827872.222956:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827872.222956:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827872.990919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827873.246978:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827873.246979:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827873.247027:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827874.014930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827874.014978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827874.015007:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827874.015013:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827874.015025:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2813 00000100:00080000:1.0:1787827874.015040:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827874.015048:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827874.015099:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564202240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827874.015150:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827874.015155:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827874.015158:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827874.015165:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2813 00000100:00080000:1.0:1787827874.015173:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827874.015180:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827874.015196:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827874.015202:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564202240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827874.015218:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564202368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827874.015231:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827874.015235:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827874.015238:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827874.015242:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2813 00000100:00080000:1.0:1787827874.015246:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827874.015249:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827874.015253:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827874.015267:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564202496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827874.015276:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827874.015280:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827874.015281:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827874.015285:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2813 00000100:00080000:1.0:1787827874.015291:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827874.015294:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827874.015299:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787827874.015309:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827874.015314:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564202624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827874.015330:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827874.015334:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827874.015336:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827874.015339:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2813 00000100:00080000:1.0:1787827874.015342:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827874.015344:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827874.015349:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827874.015358:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564202752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827874.015392:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827874/real 1787827874] req@ffff976889efcb40 x1874670564202240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827874.015414:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564202240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827874.015426:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827874.015430:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827874.015434:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564202240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827874.015462:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564202368:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827874.015488:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827874.015500:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827874/real 1787827874] req@ffff976889efe1c0 x1874670564202368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827874.015512:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564202368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827874.015520:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827874.015523:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827874.015526:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564202368:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827874.015533:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564202496:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827874.015548:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827874.015556:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827874/real 1787827874] req@ffff976889efe580 x1874670564202496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827874.015573:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564202496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827874.015581:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827874.015583:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827874.015586:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564202496:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827874.015600:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564202624:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827874.015614:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827874.015622:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827874/real 1787827874] req@ffff976889eff480 x1874670564202624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827874.015632:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564202624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827874.015639:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827874.015642:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827874.015645:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564202624:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827874.015650:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564202752:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827874.015663:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827874.015671:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827874/real 1787827874] req@ffff976889efc780 x1874670564202752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827874.015681:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564202752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827874.015693:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827874.015696:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827874.015699:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564202752:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827874.270899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827874.270907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827874.270944:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827875.038919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827875.294974:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827875.294978:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827875.295028:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827876.062977:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827876.318935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827876.318939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827876.319170:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827877.086947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827877.342929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827877.342929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827877.342997:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827878.110912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827878.367022:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827878.367032:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827878.367182:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827879.134965:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827879.135013:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827879.135027:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827879.135036:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827879.135051:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2819 00000100:00080000:0.0:1787827879.135069:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827879.135103:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827879.135190:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564202880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827879.135247:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827879.135254:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827879.135258:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827879.135269:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2819 00000100:00080000:0.0:1787827879.135279:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827879.135289:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827879.135300:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564202880:192.168.203.139@tcp:250:kworker.0 00000008:00100000:0.0:1787827879.135313:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827879.135351:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564203008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827879.135377:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827879.135384:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827879.135388:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827879.135398:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2819 00000100:00080000:0.0:1787827879.135406:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827879.135416:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787827879.135422:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827879.135437:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827879.135464:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564203136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827879.135486:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827879.135495:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827879.135499:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827879.135507:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827879/real 1787827879] req@ffff976882aef0c0 x1874670564202880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827879.135515:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2819 00000100:00080000:0.0:1787827879.135530:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827879.135539:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827879.135548:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827879.135554:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564202880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827879.135573:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564203264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827879.135599:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827879.135601:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827879.135616:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827879.135620:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:0.0:1787827879.135623:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827879.135636:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564202880:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787827879.135639:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2819 00000100:00080000:0.0:1787827879.135652:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827879.135663:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827879.135673:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827879.135697:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564203392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827879.135710:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564203008:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827879.135762:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827879.135785:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827879/real 1787827879] req@ffff976882aee940 x1874670564203008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827879.135815:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564203008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827879.135878:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827879.135886:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827879.135894:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564203008:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827879.135908:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564203136:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827879.135947:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827879.135965:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827879/real 1787827879] req@ffff976882aed680 x1874670564203136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827879.135982:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564203136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827879.135993:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827879.135998:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827879.136002:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564203136:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827879.136010:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564203264:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827879.136032:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827879.136045:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827879/real 1787827879] req@ffff976882aee580 x1874670564203264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827879.136059:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564203264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827879.136079:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827879.136112:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827879.136117:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564203264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827879.136144:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564203392:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827879.136174:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827879.136188:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827879/real 1787827879] req@ffff976882aec3c0 x1874670564203392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827879.136203:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564203392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827879.136214:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827879.136219:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827879.136223:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564203392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827879.390963:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827879.391003:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827879.391046:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827880.158918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827880.414916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827880.414933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827880.414972:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827881.182911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827881.438917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827881.438923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827881.438979:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827882.206871:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827882.462900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827882.462905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827882.462985:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827883.230898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827883.486917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827883.486917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827883.486985:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827884.254910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827884.254920:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827884.254926:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827884.254929:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827884.254937:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2824 00000100:00080000:0.0:1787827884.254946:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827884.254954:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827884.254993:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564203520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827884.255019:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827884.255023:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827884.255025:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827884.255030:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2824 00000100:00080000:0.0:1787827884.255033:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827884.255037:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827884.255043:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564203520:192.168.203.139@tcp:250:kworker.0 00000008:00100000:0.0:1787827884.255058:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827884.255077:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564203648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827884.255086:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827884.255088:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827884.255090:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827884.255093:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2824 00000100:00080000:0.0:1787827884.255104:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827884.255108:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827884.255112:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787827884.255117:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827884.255129:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564203776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827884.255140:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827884.255144:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827884.255146:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827884.255152:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827884/real 1787827884] req@ffff976882aeed00 x1874670564203520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827884.255153:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2824 00000100:00080000:0.0:1787827884.255160:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827884.255166:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827884.255171:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827884.255171:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564203520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827884.255184:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787827884.255188:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564203904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827884.255190:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827884.255198:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564203520:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787827884.255206:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827884.255210:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827884.255211:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827884.255215:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2824 00000100:00080000:0.0:1787827884.255217:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827884.255220:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827884.255223:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827884.255227:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564203648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827884.255234:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564204032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827884.255247:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827884.255260:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827884/real 1787827884] req@ffff976882aec3c0 x1874670564203648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827884.255271:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564203648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827884.255277:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827884.255280:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827884.255283:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564203648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827884.255288:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564203776:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827884.255299:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827884.255304:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827884/real 1787827884] req@ffff976882aee580 x1874670564203776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827884.255312:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564203776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827884.255317:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827884.255319:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827884.255321:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564203776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827884.255325:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564203904:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827884.255335:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827884.255340:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827884/real 1787827884] req@ffff976882aed680 x1874670564203904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827884.255348:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564203904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827884.255353:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827884.255355:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827884.255357:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564203904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827884.255366:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564204032:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827884.255376:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827884.255382:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827884/real 1787827884] req@ffff976882aee940 x1874670564204032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827884.255389:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564204032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827884.255395:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827884.255399:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827884.255402:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564204032:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827884.510888:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827884.510895:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827884.510983:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827885.278904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827885.534938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827885.534966:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827885.535005:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827886.302923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827886.558893:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827886.558897:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827886.558955:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827887.326889:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827887.582898:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827887.582898:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827887.582939:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827888.350911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827888.606900:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827888.606900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827888.606957:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827889.374904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827889.374916:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827889.374922:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827889.374926:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827889.374935:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2829 00000100:00080000:0.0:1787827889.374944:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827889.374949:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827889.374982:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564204160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827889.375002:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827889.375005:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827889.375006:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827889.375022:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2829 00000100:00080000:0.0:1787827889.375026:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827889.375029:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827889.375038:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827889.375038:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564204160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827889.375056:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564204288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827889.375077:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827889.375081:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827889.375084:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827889.375088:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2829 00000100:00080000:0.0:1787827889.375093:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827889.375097:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787827889.375102:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827889.375110:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827889.375125:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564204416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827889.375133:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827889/real 1787827889] req@ffff976882aef0c0 x1874670564204160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827889.375145:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827889.375150:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827889.375152:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827889.375155:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564204160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827889.375160:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2829 00000100:00080000:0.0:1787827889.375167:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827889.375171:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827889.375173:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:3.0:1787827889.375179:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000008:00100000:0.0:1787827889.375180:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827889.375187:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564204160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827889.375194:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564204544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827889.375202:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827889.375204:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827889.375205:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827889.375208:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2829 00000100:00080000:0.0:1787827889.375209:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827889.375212:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827889.375216:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827889.375219:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564204288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827889.375226:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564204672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827889.375253:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827889.375267:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827889/real 1787827889] req@ffff976882aee940 x1874670564204288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827889.375280:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564204288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827889.375289:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827889.375293:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827889.375297:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564204288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827889.375303:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564204416:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827889.375325:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827889.375336:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827889/real 1787827889] req@ffff976882aed680 x1874670564204416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827889.375350:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564204416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827889.375359:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827889.375363:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827889.375366:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564204416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827889.375372:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564204544:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827889.375392:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827889.375404:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827889/real 1787827889] req@ffff976882aee580 x1874670564204544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827889.375416:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564204544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827889.375425:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827889.375428:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827889.375431:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564204544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827889.375449:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564204672:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827889.375470:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827889.375481:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827889/real 1787827889] req@ffff976882aec3c0 x1874670564204672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827889.375494:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564204672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827889.375509:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827889.375512:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827889.375516:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564204672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827889.630895:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827889.630901:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827889.630951:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827890.398907:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827890.654908:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827890.654908:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827890.654931:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827891.422902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827891.678910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827891.678942:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827891.678953:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827892.446903:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827892.702883:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827892.702921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827892.702974:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827893.470875:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827893.726918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827893.726919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827893.726952:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827894.494879:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827894.494899:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827894.494907:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827894.494912:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827894.494921:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2834 00000100:00080000:0.0:1787827894.494931:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827894.494939:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827894.494986:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564204800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827894.495012:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827894.495016:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827894.495018:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827894.495023:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2834 00000100:00080000:0.0:1787827894.495027:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827894.495032:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827894.495042:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827894.495048:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564204800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827894.495062:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564204928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827894.495076:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827894.495080:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827894.495082:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827894.495086:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2834 00000100:00080000:0.0:1787827894.495091:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827894.495095:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827894.495101:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787827894.495106:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827894.495118:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564205056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827894.495131:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827894.495134:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827894.495136:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827894.495139:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827894/real 1787827894] req@ffff976882aeed00 x1874670564204800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827894.495145:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2834 00000100:00080000:0.0:1787827894.495153:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827894.495160:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827894.495163:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564204800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827894.495168:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1787827894.495173:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827894.495177:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827894.495180:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564204800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827894.495185:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564205184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827894.495197:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827894.495202:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564204928:192.168.203.139@tcp:38:kworker.0 00000100:00080000:0.0:1787827894.495203:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827894.495208:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827894.495214:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2834 00000100:00080000:0.0:1787827894.495219:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827894.495225:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827894.495236:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827894/real 1787827894] req@ffff976882aec3c0 x1874670564204928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827894.495238:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827894.495250:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827894.495252:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564204928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827894.495261:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827894.495264:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827894.495268:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564205312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827894.495270:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564204928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827894.495280:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564205056:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827894.495301:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827894.495310:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827894/real 1787827894] req@ffff976882aee580 x1874670564205056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827894.495318:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564205056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827894.495323:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827894.495325:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827894.495328:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564205056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827894.495331:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564205184:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827894.495341:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827894.495347:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827894/real 1787827894] req@ffff976882aed680 x1874670564205184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827894.495354:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564205184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827894.495359:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827894.495361:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827894.495364:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564205184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827894.495377:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564205312:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827894.495396:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827894.495408:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827894/real 1787827894] req@ffff976882aee940 x1874670564205312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827894.495421:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564205312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827894.495430:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827894.495432:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827894.495435:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564205312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827894.750870:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827894.750906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827894.750950:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827895.518902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827895.774876:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827895.774900:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827895.774919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787827896.542927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827896.798892:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827896.798934:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827896.798945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827897.566906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827897.822922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827897.822922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827897.822948:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827898.590929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827898.846935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827898.846936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827898.847016:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827899.614955:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827899.614955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827899.614969:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827899.614974:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827899.614986:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2839 00000100:00080000:1.0:1787827899.614998:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827899.615006:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827899.615049:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564205440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827899.615078:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827899.615081:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827899.615083:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827899.615088:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2839 00000100:00080000:1.0:1787827899.615094:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827899.615098:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827899.615109:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827899.615118:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564205440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827899.615125:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564205568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827899.615137:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827899.615140:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827899.615143:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827899.615148:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2839 00000100:00080000:1.0:1787827899.615151:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827899.615157:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827899.615165:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827899.615175:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564205696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827899.615184:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827899.615187:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827899.615188:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827899.615191:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2839 00000100:00080000:1.0:1787827899.615194:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827899.615198:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:0.0:1787827899.615204:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787827899.615205:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827899.615216:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564205824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827899.615223:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827899.615226:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827899.615227:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827899.615231:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2839 00000100:00080000:1.0:1787827899.615233:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827899.615235:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827899.615239:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827899.615249:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827899/real 1787827899] req@ffff976889efc780 x1874670564205440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827899.615250:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564205952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827899.615305:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564205440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827899.615316:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827899.615319:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827899.615323:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564205440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827899.615345:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564205568:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827899.615362:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827899.615372:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827899/real 1787827899] req@ffff976889eff480 x1874670564205568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827899.615381:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564205568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827899.615388:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827899.615391:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827899.615393:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564205568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827899.615397:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564205696:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827899.615408:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827899.615415:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827899/real 1787827899] req@ffff976889efe580 x1874670564205696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827899.615422:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564205696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827899.615430:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827899.615434:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827899.615437:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564205696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827899.615443:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564205824:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827899.615466:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827899.615478:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827899/real 1787827899] req@ffff976889efe1c0 x1874670564205824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827899.615490:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564205824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827899.615499:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827899.615502:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827899.615506:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564205824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827899.615512:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564205952:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827899.615532:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827899.615542:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827899/real 1787827899] req@ffff976889efcb40 x1874670564205952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827899.615554:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564205952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827899.615562:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827899.615566:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827899.615569:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564205952:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827899.870920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827899.870928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827899.870933:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827900.638898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827900.894922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827900.894960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827900.894990:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827901.662890:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827901.918896:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827901.918915:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827901.918967:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787827902.686890:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827902.942972:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827902.942981:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827902.942989:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827903.710900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827903.966929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827903.966929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827903.966939:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827904.734950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827904.734962:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827904.734973:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827904.734980:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827904.734994:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2844 00000100:00080000:1.0:1787827904.735010:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827904.735019:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827904.735068:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564206080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827904.735096:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827904.735099:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827904.735100:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827904.735103:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2844 00000100:00080000:1.0:1787827904.735106:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827904.735109:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827904.735119:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827904.735128:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564206208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827904.735137:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827904.735140:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827904.735141:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827904.735147:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564206080:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827904.735147:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2844 00000100:00080000:1.0:1787827904.735153:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827904.735157:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827904.735161:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827904.735169:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564206336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827904.735181:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827904.735184:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827904.735185:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827904.735188:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2844 00000100:00080000:1.0:1787827904.735190:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827904.735192:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827904.735196:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827904.735204:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564206464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827904.735209:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827904.735212:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827904.735213:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827904.735216:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2844 00000100:00080000:1.0:1787827904.735218:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827904.735220:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:0.0:1787827904.735225:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787827904.735226:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827904.735235:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564206592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827904.735279:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827904/real 1787827904] req@ffff976889eff0c0 x1874670564206080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827904.735302:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564206080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827904.735322:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827904.735326:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827904.735329:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564206080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827904.735355:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564206208:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827904.735377:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827904.735388:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827904/real 1787827904] req@ffff976889efed00 x1874670564206208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827904.735397:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564206208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827904.735403:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827904.735405:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827904.735408:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564206208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827904.735413:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564206336:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827904.735432:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827904.735444:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827904/real 1787827904] req@ffff976889efc3c0 x1874670564206336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827904.735467:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564206336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827904.735473:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827904.735476:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827904.735478:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564206336:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827904.735483:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564206464:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827904.735500:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827904.735508:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827904/real 1787827904] req@ffff976889efe940 x1874670564206464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827904.735520:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564206464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827904.735525:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827904.735527:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827904.735529:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564206464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827904.735532:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564206592:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827904.735543:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827904.735548:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827904/real 1787827904] req@ffff976889effc00 x1874670564206592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827904.735556:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564206592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827904.735561:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827904.735563:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827904.735564:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564206592:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827904.990929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827904.990980:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827904.990986:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827905.758998:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827906.014934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827906.014934:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827906.014966:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827906.782931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827907.038960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787827907.038960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827907.039067:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827907.806928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827908.062965:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827908.062970:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827908.063049:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827908.830898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827909.086906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827909.086909:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827909.086965:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827909.854953:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827909.854958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827909.854966:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827909.854973:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827909.854985:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2849 00000100:00080000:1.0:1787827909.854999:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827909.855008:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827909.855062:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564206720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827909.855090:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827909.855105:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827909.855108:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827909.855113:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2849 00000100:00080000:1.0:1787827909.855117:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827909.855121:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827909.855134:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827909.855148:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564206848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827909.855153:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564206720:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827909.855159:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827909.855162:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827909.855164:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827909.855168:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2849 00000100:00080000:1.0:1787827909.855171:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827909.855175:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827909.855179:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827909.855190:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564206976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827909.855204:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827909.855207:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827909.855208:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827909.855213:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2849 00000100:00080000:1.0:1787827909.855217:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827909.855221:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827909.855225:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827909.855236:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564207104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827909.855245:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827909.855248:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827909.855250:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827909.855254:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2849 00000100:00080000:1.0:1787827909.855257:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827909.855260:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827909.855265:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827909.855274:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564207232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827909.855279:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564206848:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827909.855312:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564206976:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827909.855357:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564207104:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827909.855379:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564207232:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827910.110936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827910.110936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827910.111023:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827910.878910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827911.134923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827911.134958:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827911.134966:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827911.902965:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827912.158971:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827912.158971:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827912.159042:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827912.926908:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:2.0:1787827912.927055:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:2.0:1787827912.927086:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787827912.927098:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787827912.927128:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787827912.927134:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787827912.927140:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787827912.927146:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:0.0:1787827912.927215:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827909/real 1787827912] req@ffff976889eff0c0 x1874670564207232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827912.927247:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564207232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827912.927258:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827912.927262:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827912.927266:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564207232:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827912.927273:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827909/real 1787827912] req@ffff976889efe940 x1874670564206848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827912.927281:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564206848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827912.927295:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827912.927301:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827912.927304:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564206848:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827912.927310:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827909/real 1787827912] req@ffff976889efc3c0 x1874670564206976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827912.927328:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564206976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827912.927337:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827912.927342:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827912.927346:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564206976:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827912.927354:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827909/real 1787827912] req@ffff976889efed00 x1874670564207104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827912.927368:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564207104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827912.927376:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827912.927379:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827912.927382:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564207104:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827912.927389:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827909/real 1787827912] req@ffff976889effc00 x1874670564206720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827912.927405:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564206720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827912.927414:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827912.927417:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827912.927421:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564206720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827913.182922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827913.182922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827913.182944:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827913.950938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827914.206922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827914.206922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827914.206976:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827914.974936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827914.974964:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827914.974970:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827914.974974:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827914.974982:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2854 00000100:00080000:0.0:1787827914.974990:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827914.974996:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827914.975025:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564207360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827914.975049:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827914.975051:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827914.975053:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827914.975055:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2854 00000100:00080000:0.0:1787827914.975058:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827914.975060:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827914.975069:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827914.975078:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564207488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827914.975082:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564207360:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787827914.975094:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827914.975096:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827914.975097:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827914.975100:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2854 00000100:00080000:0.0:1787827914.975102:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827914.975104:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827914.975109:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827914.975116:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564207616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827914.975123:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827914.975125:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827914.975127:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827914.975129:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2854 00000100:00080000:0.0:1787827914.975132:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827914.975134:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827914.975137:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827914.975144:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564207744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827914.975150:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827914.975154:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1787827914.975155:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787827914.975156:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827914.975163:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2854 00000100:00080000:0.0:1787827914.975167:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827914.975169:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827914.975172:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827914.975180:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564207872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827914.975204:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827914/real 1787827914] req@ffff976882aef0c0 x1874670564207360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827914.975224:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564207360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827914.975234:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827914.975239:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827914.975243:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564207360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827914.975269:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564207488:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827914.975296:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827914.975308:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827914/real 1787827914] req@ffff976882aee940 x1874670564207488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827914.975327:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564207488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827914.975336:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827914.975339:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827914.975342:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564207488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827914.975348:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564207616:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827914.975368:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827914.975379:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827914/real 1787827914] req@ffff976882aed680 x1874670564207616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827914.975392:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564207616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827914.975400:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827914.975403:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827914.975407:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564207616:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827914.975413:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564207744:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827914.975434:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827914.975446:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827914/real 1787827914] req@ffff976882aee580 x1874670564207744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827914.975459:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564207744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827914.975470:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827914.975472:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827914.975474:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564207744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827914.975481:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564207872:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827914.975510:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827914.975522:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827914/real 1787827914] req@ffff976882aec3c0 x1874670564207872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827914.975534:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564207872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827914.975542:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827914.975545:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827914.975548:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564207872:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827915.230923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827915.230938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827915.230977:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827915.998888:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827916.254946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827916.254946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827916.254960:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827917.022899:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827917.278982:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827917.278983:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827917.279005:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827918.046927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827918.302953:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827918.302985:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827918.303026:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827919.070919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787827919.326931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827919.326938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827919.326986:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827920.094889:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827920.094924:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827920.094932:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827920.094936:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827920.094947:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2859 00000100:00080000:2.0:1787827920.094959:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827920.094966:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827920.095011:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564208000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827920.095039:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827920.095042:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827920.095043:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827920.095047:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2859 00000100:00080000:2.0:1787827920.095050:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827920.095053:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827920.095063:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827920.095072:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564208128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827920.095081:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827920.095084:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564208000:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787827920.095085:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827920.095091:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827920.095105:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2859 00000100:00080000:2.0:1787827920.095108:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827920.095111:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827920.095113:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827920.095126:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564208256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827920.095132:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827920.095142:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827920.095144:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827920.095148:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2859 00000100:00080000:2.0:1787827920.095157:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827920.095162:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827920.095166:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827920.095178:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564208384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827920.095182:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787827920.095200:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827920.095203:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827920.095204:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827920.095209:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2859 00000100:00080000:2.0:1787827920.095211:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827920.095216:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827920.095221:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827920.095231:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564208512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827920.095241:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827920/real 1787827920] req@ffff976889cfa580 x1874670564208000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827920.095265:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564208000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827920.095277:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827920.095282:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827920.095286:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564208000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827920.095310:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564208128:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827920.095335:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827920.095347:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827920/real 1787827920] req@ffff976889cfa940 x1874670564208128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827920.095360:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564208128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827920.095370:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827920.095374:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827920.095377:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564208128:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827920.095384:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564208256:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827920.095400:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827920.095409:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827920/real 1787827920] req@ffff976889cfbc00 x1874670564208256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827920.095428:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564208256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827920.095435:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827920.095438:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827920.095441:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564208256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827920.095447:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564208384:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827920.095466:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827920.095475:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827920/real 1787827920] req@ffff976889cf8000 x1874670564208384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827920.095486:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564208384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827920.095494:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827920.095497:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827920.095503:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564208384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827920.095510:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564208512:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827920.095540:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827920.095551:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827920/real 1787827920] req@ffff976889cfb0c0 x1874670564208512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827920.095565:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564208512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827920.095582:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827920.095588:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827920.095592:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564208512:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827920.350943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827920.350951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827920.350955:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827921.118902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827921.374913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827921.374960:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827921.375039:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827922.142914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827922.398934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827922.398934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827922.398990:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827923.167058:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827923.422970:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827923.422982:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827923.423045:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827924.190897:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827924.446981:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827924.446981:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827924.447069:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827925.214974:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827925.214986:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827925.214987:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827925.214995:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827925.215009:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2865 00000100:00080000:2.0:1787827925.215023:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827925.215032:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827925.215097:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564208640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827925.215160:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827925.215165:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827925.215167:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827925.215182:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2865 00000100:00080000:2.0:1787827925.215187:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827925.215191:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827925.215203:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827925.215221:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564208768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827925.215234:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564208640:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787827925.215270:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827925.215283:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827925.215285:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827925.215290:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2865 00000100:00080000:2.0:1787827925.215293:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827925.215296:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827925.215302:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827925.215331:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564208896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827925.215342:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827925.215346:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827925.215348:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827925.215352:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2865 00000100:00080000:2.0:1787827925.215356:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827925.215362:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787827925.215368:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827925.215376:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787827925.215392:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564209024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827925.215403:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827925.215406:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827925.215409:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827925.215414:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2865 00000100:00100000:3.0:1787827925.215443:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827925/real 1787827925] req@ffff976889cf8f00 x1874670564208640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827925.215445:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827925.215454:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827925.215463:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827925.215477:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564208640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827925.215478:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564209152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827925.215538:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827925.215546:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827925.215553:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564208640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827925.215619:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564208768:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827925.215671:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827925.215698:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827925/real 1787827925] req@ffff976889cfb0c0 x1874670564208768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827925.215730:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564208768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827925.215750:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827925.215757:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827925.215793:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564208768:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827925.215894:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564208896:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827925.215950:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827925.215967:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827925/real 1787827925] req@ffff976889cf8000 x1874670564208896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827925.215989:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564208896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827925.215997:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827925.216000:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827925.216020:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564208896:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827925.216027:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564209024:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827925.216053:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827925.216065:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827925/real 1787827925] req@ffff976889cfbc00 x1874670564209024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827925.216077:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564209024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827925.216086:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827925.216089:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827925.216094:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564209024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827925.216100:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564209152:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827925.216116:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827925.216126:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827925/real 1787827925] req@ffff976889cfa940 x1874670564209152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827925.216137:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564209152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827925.216146:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827925.216148:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827925.216152:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564209152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827925.470917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827925.470932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827925.470992:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827926.239001:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827926.495079:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827926.495122:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827926.495140:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827927.262918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827927.518948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827927.518976:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827927.519035:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827928.286931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827928.542971:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827928.543019:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827928.543050:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827929.310941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787827929.567030:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827929.567039:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827929.567042:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827930.334934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827930.334974:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827930.334981:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827930.334986:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827930.334998:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2870 00000100:00080000:1.0:1787827930.335011:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827930.335019:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827930.335069:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564209280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827930.335100:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827930.335103:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827930.335104:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827930.335108:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2870 00000100:00080000:1.0:1787827930.335110:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827930.335113:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827930.335131:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827930.335143:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564209408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827930.335150:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827930.335153:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827930.335155:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827930.335158:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2870 00000100:00080000:1.0:1787827930.335161:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827930.335163:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827930.335167:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827930.335176:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564209536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827930.335184:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827930.335188:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787827930.335188:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564209280:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827930.335189:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827930.335198:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2870 00000100:00080000:1.0:1787827930.335201:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827930.335203:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827930.335206:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827930.335214:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564209664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827930.335221:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827930.335223:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827930.335225:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827930.335227:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2870 00000100:00080000:1.0:1787827930.335229:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827930.335232:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827930.335235:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827930.335242:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564209792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827930.335274:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827930.335330:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827930/real 1787827930] req@ffff976889efcb40 x1874670564209280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827930.335347:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564209280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827930.335357:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827930.335360:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827930.335363:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564209280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827930.335371:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564209408:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827930.335387:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827930.335396:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827930/real 1787827930] req@ffff976889efe1c0 x1874670564209408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827930.335405:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564209408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827930.335411:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827930.335413:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827930.335416:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564209408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827930.335442:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564209536:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787827930.335453:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827930.335459:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827930/real 1787827930] req@ffff976889efe580 x1874670564209536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827930.335466:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564209536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827930.335473:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827930.335474:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827930.335477:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564209536:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787827930.335481:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564209664:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827930.335491:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827930.335497:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827930/real 1787827930] req@ffff976889eff480 x1874670564209664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827930.335505:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564209664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827930.335511:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827930.335512:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827930.335514:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564209664:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787827930.335517:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564209792:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787827930.335528:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827930.335534:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827930/real 1787827930] req@ffff976889efc780 x1874670564209792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827930.335541:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564209792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827930.335546:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827930.335548:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787827930.335550:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564209792:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827930.591057:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827930.591058:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827930.591195:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827931.358899:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827931.614924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827931.614925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827931.614953:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827932.382932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827932.638881:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827932.638898:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827932.638947:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827933.406973:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827933.662949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827933.662972:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827933.663018:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827934.430954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827934.686986:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827934.686986:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827934.687160:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827935.454926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827935.454998:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827935.455015:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827935.455025:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827935.455044:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2875 00000100:00080000:1.0:1787827935.455065:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827935.455077:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827935.455143:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564209920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827935.455193:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827935.455198:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827935.455201:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827935.455208:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2875 00000100:00080000:1.0:1787827935.455215:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827935.455222:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827935.455239:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827935.455250:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564209920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827935.455264:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564210048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827935.455289:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827935.455297:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827935.455301:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827935.455311:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2875 00000100:00080000:1.0:1787827935.455338:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787827935.455350:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787827935.455354:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827935.455366:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827935.455395:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564210176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827935.455414:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827935.455420:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827935/real 1787827935] req@ffff976889efc780 x1874670564209920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827935.455421:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827935.455431:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827935.455451:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2875 00000100:00080000:1.0:1787827935.455462:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827935.455474:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564209920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827935.455480:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827935.455502:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00080000:0.0:1787827935.455510:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827935.455517:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827935.455524:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564210304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827935.455534:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564209920:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827935.455551:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827935.455556:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827935.455560:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827935.455569:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2875 00000100:00080000:1.0:1787827935.455573:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827935.455578:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827935.455584:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827935.455603:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564210048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827935.455605:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564210432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827935.455681:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827935.455735:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827935/real 1787827935] req@ffff976889eff480 x1874670564210048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827935.455753:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564210048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827935.455769:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827935.455774:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827935.455782:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564210048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827935.455794:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564210176:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827935.455880:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827935.455905:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827935/real 1787827935] req@ffff976889efe580 x1874670564210176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827935.455934:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564210176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827935.455951:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827935.455955:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827935.455960:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564210176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827935.455971:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564210304:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827935.456003:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827935.456023:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827935/real 1787827935] req@ffff976889efe1c0 x1874670564210304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827935.456050:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564210304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827935.456059:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827935.456063:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827935.456066:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564210304:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827935.456088:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564210432:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827935.456111:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827935.456130:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827935/real 1787827935] req@ffff976889efcb40 x1874670564210432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827935.456141:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564210432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827935.456149:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827935.456153:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827935.456156:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564210432:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827935.710948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827935.710957:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827935.711128:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827936.478925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827936.734965:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827936.735047:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827936.735189:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827937.502922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827937.758927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827937.758949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827937.758954:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827938.526972:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827938.782918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827938.782919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827938.783005:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827939.550957:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827939.806938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827939.807008:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827939.807028:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827940.574888:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827940.574899:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827940.574905:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827940.574910:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827940.574920:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2880 00000100:00080000:3.0:1787827940.574930:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827940.574937:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827940.574976:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564210560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827940.575003:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827940.575006:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827940.575008:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827940.575013:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2880 00000100:00080000:3.0:1787827940.575018:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827940.575023:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827940.575030:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564210560:192.168.203.139@tcp:250:kworker.0 00000008:00100000:3.0:1787827940.575033:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827940.575054:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564210688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827940.575063:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827940.575065:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827940.575067:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827940.575070:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2880 00000100:00080000:3.0:1787827940.575073:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827940.575075:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827940.575079:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827940.575087:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564210816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827940.575102:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827940.575104:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827940.575105:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787827940.575108:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827940.575109:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2880 00000100:00080000:3.0:1787827940.575114:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827940.575117:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827940.575120:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827940.575128:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564210944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827940.575134:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827940.575136:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827940.575137:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827940.575140:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2880 00000100:00080000:3.0:1787827940.575142:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827940.575145:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827940.575148:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827940.575152:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827940/real 1787827940] req@ffff97688cb5de00 x1874670564210560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827940.575158:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564211072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827940.575183:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564210560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827940.575193:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827940.575197:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827940.575201:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564210560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827940.575226:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564210688:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827940.575252:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827940.575267:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827940/real 1787827940] req@ffff97688cb5f0c0 x1874670564210688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827940.575280:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564210688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827940.575289:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827940.575293:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827940.575297:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564210688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827940.575304:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564210816:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827940.575325:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827940.575337:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827940/real 1787827940] req@ffff97688cb5d680 x1874670564210816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827940.575346:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564210816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827940.575351:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827940.575354:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827940.575356:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564210816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827940.575360:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564210944:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827940.575374:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827940.575380:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827940/real 1787827940] req@ffff97688cb5c000 x1874670564210944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827940.575387:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564210944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827940.575392:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827940.575394:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827940.575397:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564210944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827940.575400:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564211072:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827940.575410:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827940.575415:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827940/real 1787827940] req@ffff97688cb5cb40 x1874670564211072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827940.575422:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564211072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827940.575427:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827940.575429:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827940.575431:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564211072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827940.830904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827940.830906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827940.830976:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827941.598913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827941.854943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827941.855012:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827941.855127:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827942.623055:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787827942.879038:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827942.879107:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827942.879176:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787827943.646951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787827943.902913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827943.902951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827943.903027:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827944.670911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827944.926924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827944.926968:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827944.927034:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827945.694899:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827945.694949:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827945.694957:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827945.694963:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827945.694974:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2885 00000100:00080000:2.0:1787827945.694987:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827945.694994:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827945.695040:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564211200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827945.695069:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827945.695072:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827945.695073:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827945.695077:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2885 00000100:00080000:2.0:1787827945.695079:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827945.695081:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827945.695092:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827945.695101:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564211328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827945.695107:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827945.695111:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827945.695112:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827945.695114:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2885 00000100:00080000:2.0:1787827945.695116:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827945.695118:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827945.695120:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827945.695126:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564211456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827945.695137:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827945.695139:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827945.695141:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827945.695144:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2885 00000100:00080000:2.0:1787827945.695146:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827945.695148:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827945.695152:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827945.695161:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564211200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787827945.695161:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564211584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827945.695175:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827945.695177:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827945.695179:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827945.695181:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2885 00000100:00080000:2.0:1787827945.695183:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827945.695185:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827945.695188:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827945.695196:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564211712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827945.695242:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827945.695287:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827945/real 1787827945] req@ffff976889cfa580 x1874670564211200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827945.695309:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564211200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827945.695318:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827945.695321:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827945.695324:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564211200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827945.695332:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564211328:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827945.695347:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827945.695356:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827945/real 1787827945] req@ffff976889cfa1c0 x1874670564211328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827945.695364:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564211328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827945.695370:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827945.695373:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827945.695375:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564211328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827945.695378:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564211456:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827945.695388:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827945.695393:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827945/real 1787827945] req@ffff976889cf92c0 x1874670564211456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827945.695401:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564211456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827945.695406:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827945.695408:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827945.695410:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564211456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827945.695433:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564211584:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827945.695444:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827945.695451:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827945/real 1787827945] req@ffff976889cf9e00 x1874670564211584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827945.695459:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564211584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827945.695464:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827945.695466:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827945.695468:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564211584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827945.695471:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564211712:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827945.695481:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827945.695487:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827945/real 1787827945] req@ffff976889cfb480 x1874670564211712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827945.695494:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564211712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827945.695499:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827945.695501:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827945.695503:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564211712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827945.950954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827945.950975:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827945.951018:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827946.718917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827946.974918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827946.974942:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827946.974969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787827947.742907:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827947.998908:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827947.998919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827947.999002:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827948.766893:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827949.022963:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827949.022974:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827949.023031:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827949.790915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827950.046932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827950.046955:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827950.047035:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827950.814961:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827950.814966:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827950.814972:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827950.814978:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827950.814989:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2890 00000100:00080000:3.0:1787827950.815001:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827950.815008:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827950.815052:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564211840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827950.815078:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827950.815082:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827950.815084:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827950.815088:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2890 00000100:00080000:3.0:1787827950.815092:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827950.815097:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827950.815110:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827950.815123:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564211968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827950.815131:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564211840:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827950.815134:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827950.815137:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827950.815140:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827950.815143:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2890 00000100:00080000:3.0:1787827950.815147:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827950.815150:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827950.815154:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827950.815163:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564212096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827950.815173:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827950.815176:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827950.815178:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827950.815182:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2890 00000100:00080000:3.0:1787827950.815185:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827950.815189:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827950.815193:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827950.815203:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564212224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827950.815211:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827950.815214:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827950.815216:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787827950.815217:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787827950.815220:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2890 00000100:00080000:3.0:1787827950.815222:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827950.815226:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827950.815230:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827950.815242:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564212352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827950.815267:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827950/real 1787827950] req@ffff97688cb5f840 x1874670564211840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827950.815294:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564211840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827950.815314:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827950.815319:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827950.815322:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564211840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827950.815350:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564211968:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827950.815370:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827950.815381:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827950/real 1787827950] req@ffff97688cb5c3c0 x1874670564211968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827950.815390:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564211968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827950.815396:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827950.815398:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827950.815401:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564211968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827950.815404:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564212096:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827950.815415:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827950.815421:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827950/real 1787827950] req@ffff97688cb5e1c0 x1874670564212096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827950.815429:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564212096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827950.815439:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827950.815441:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827950.815443:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564212096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827950.815446:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564212224:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827950.815458:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827950.815464:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827950/real 1787827950] req@ffff97688cb5c780 x1874670564212224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827950.815471:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564212224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827950.815476:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827950.815478:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827950.815480:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564212224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827950.815483:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564212352:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827950.815493:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827950.815498:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827950/real 1787827950] req@ffff97688cb5cf00 x1874670564212352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827950.815505:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564212352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827950.815510:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827950.815512:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827950.815514:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564212352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827951.070981:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827951.070981:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827951.071054:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827951.838889:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827952.094902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827952.094943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827952.095044:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827952.862986:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827953.119020:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827953.119029:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787827953.119060:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827953.886948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827954.142972:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827954.143032:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827954.143040:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827954.910970:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827955.166988:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827955.166988:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827955.167097:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827955.934916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827955.934924:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827955.934932:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827955.934938:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827955.934947:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2895 00000100:00080000:1.0:1787827955.934956:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827955.934963:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827955.935007:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564212480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827955.935031:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827955.935034:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827955.935036:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827955.935039:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2895 00000100:00080000:1.0:1787827955.935042:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827955.935044:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827955.935053:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827955.935063:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564212608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827955.935071:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827955.935074:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827955.935076:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827955.935079:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2895 00000100:00080000:1.0:1787827955.935082:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827955.935085:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827955.935089:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827955.935098:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564212736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827955.935104:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827955.935107:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827955.935109:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827955.935113:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2895 00000100:00080000:1.0:1787827955.935116:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827955.935119:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827955.935124:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827955.935133:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564212480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827955.935145:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564212864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827955.935154:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827955.935157:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827955.935159:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827955.935161:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2895 00000100:00080000:1.0:1787827955.935165:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827955.935168:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827955.935172:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827955.935181:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564212992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827955.935238:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827955.935311:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827955/real 1787827955] req@ffff976889effc00 x1874670564212480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827955.935337:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564212480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827955.935348:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827955.935352:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827955.935356:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564212480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827955.935365:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564212608:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827955.935390:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827955.935401:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827955/real 1787827955] req@ffff976889efed00 x1874670564212608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827955.935411:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564212608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827955.935418:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827955.935421:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827955.935425:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564212608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827955.935430:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564212736:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827955.935443:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827955.935450:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827955/real 1787827955] req@ffff976889efc3c0 x1874670564212736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827955.935459:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564212736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827955.935465:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827955.935467:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827955.935469:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564212736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827955.935499:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564212864:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827955.935513:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827955.935520:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827955/real 1787827955] req@ffff976889efe940 x1874670564212864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827955.935529:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564212864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827955.935535:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827955.935537:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827955.935539:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564212864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827955.935544:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564212992:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827955.935577:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827955.935587:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827955/real 1787827955] req@ffff976889eff0c0 x1874670564212992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827955.935596:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564212992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827955.935603:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827955.935606:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827955.935608:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564212992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787827956.190944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827956.190978:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827956.190983:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827956.958949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827957.214909:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827957.214909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827957.214940:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827957.982961:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787827958.238949:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827958.238954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827958.238978:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827959.006895:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827959.262891:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827959.262904:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827959.262925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827960.030935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827960.286930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827960.286985:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787827960.287053:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827961.054962:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827961.054972:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827961.054977:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827961.054988:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2900 00000100:00100000:0.0:1787827961.055000:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827961.055018:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827961.055029:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827961.055065:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564213120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827961.055110:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827961.055113:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827961.055114:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827961.055118:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2900 00000100:00080000:2.0:1787827961.055121:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827961.055125:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827961.055135:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827961.055170:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564213248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827961.055171:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564213120:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787827961.055190:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827961.055193:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827961.055195:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827961.055198:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2900 00000100:00080000:2.0:1787827961.055202:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827961.055205:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827961.055209:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827961.055221:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564213376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827961.055229:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827961.055232:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827961.055234:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827961.055237:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2900 00000100:00080000:2.0:1787827961.055241:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827961.055245:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827961.055250:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1787827961.055258:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787827961.055280:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564213504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827961.055291:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827961.055295:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827961.055297:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827961.055300:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2900 00000100:00080000:2.0:1787827961.055303:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827961.055307:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827961.055313:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827961.055321:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827961/real 1787827961] req@ffff976889cfad00 x1874670564213120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827961.055327:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564213632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827961.055359:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564213120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827961.055372:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827961.055376:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827961.055379:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564213120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827961.055416:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564213248:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827961.055439:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827961.055450:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827961/real 1787827961] req@ffff976889cfb480 x1874670564213248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827961.055460:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564213248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827961.055467:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827961.055470:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827961.055473:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564213248:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827961.055477:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564213376:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827961.055489:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827961.055497:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827961/real 1787827961] req@ffff976889cf9e00 x1874670564213376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827961.055531:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564213376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827961.055539:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827961.055541:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827961.055544:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564213376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827961.055549:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564213504:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827961.055586:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827961.055596:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827961/real 1787827961] req@ffff976889cf92c0 x1874670564213504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827961.055612:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564213504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827961.055619:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827961.055639:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827961.055642:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564213504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827961.055647:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564213632:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827961.055662:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827961.055670:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827961/real 1787827961] req@ffff976889cfa1c0 x1874670564213632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827961.055680:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564213632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827961.055704:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827961.055706:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827961.055709:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564213632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827961.310899:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827961.310933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827961.310954:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827962.078982:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827962.334988:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827962.334993:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827962.335070:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827963.102926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827963.358945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827963.358971:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827963.359099:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827964.126996:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827964.382889:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827964.382897:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827964.382940:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827965.150928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827965.406939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827965.406957:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827965.407006:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827966.174993:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787827966.175009:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827966.175020:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827966.175027:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827966.175041:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2906 00000100:00080000:1.0:1787827966.175066:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827966.175075:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827966.175132:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564213760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827966.175183:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827966.175192:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827966.175195:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827966.175202:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2906 00000100:00080000:1.0:1787827966.175210:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827966.175217:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827966.175232:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827966.175246:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564213760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827966.175259:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564213888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827966.175278:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827966.175283:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827966.175286:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827966.175292:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2906 00000100:00080000:1.0:1787827966.175297:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827966.175302:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827966.175308:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827966.175327:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564214016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827966.175339:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787827966.175353:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827966.175359:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827966.175362:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827966.175369:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2906 00000100:00080000:1.0:1787827966.175374:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827966.175379:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827966.175385:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827966.175406:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827966/real 1787827966] req@ffff976889eff0c0 x1874670564213760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827966.175412:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564214144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827966.175457:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827966.175460:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564213760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827966.175508:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827966.175511:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827966.175518:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:1.0:1787827966.175521:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827966.175530:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564213760:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787827966.175539:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2906 00000100:00080000:1.0:1787827966.175549:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827966.175555:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787827966.175564:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827966.175575:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564213888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827966.175587:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564214272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827966.175629:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827966.175652:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827966/real 1787827966] req@ffff976889efe940 x1874670564213888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827966.175669:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564213888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827966.175681:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827966.175685:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827966.175690:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564213888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827966.175698:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564214016:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787827966.175720:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827966.175732:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827966/real 1787827966] req@ffff976889efc3c0 x1874670564214016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827966.175755:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564214016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827966.175765:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827966.175769:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827966.175773:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564214016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787827966.175780:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564214144:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827966.175799:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827966.175811:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827966/real 1787827966] req@ffff976889efed00 x1874670564214144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827966.175856:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564214144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827966.175867:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827966.175871:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827966.175875:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564214144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827966.175897:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564214272:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787827966.175921:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827966.175934:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827966/real 1787827966] req@ffff976889effc00 x1874670564214272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827966.175949:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564214272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827966.175967:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827966.175971:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787827966.175975:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564214272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827966.430938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827966.430948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827966.431042:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827967.198937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827967.454990:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827967.454999:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827967.455007:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827968.222938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827968.479000:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827968.479040:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827968.479135:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827969.246937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827969.502909:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827969.502909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827969.502941:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827970.270959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827970.526954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827970.527010:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827970.527128:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827971.295058:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787827971.295094:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827971.295108:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827971.295117:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827971.295135:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2911 00000100:00080000:2.0:1787827971.295153:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827971.295167:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787827971.295223:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564214400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827971.295275:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827971.295284:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827971.295290:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827971.295303:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2911 00000100:00080000:2.0:1787827971.295313:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827971.295320:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564214400:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787827971.295330:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827971.295353:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827971.295387:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564214528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827971.295407:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787827971.295417:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787827971.295420:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827971.295425:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827971.295433:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2911 00000100:00080000:2.0:1787827971.295468:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827971.295479:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787827971.295492:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827971.295503:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827971/real 1787827971] req@ffff976889cfa580 x1874670564214400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827971.295523:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564214656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827971.295537:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564214400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827971.295585:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827971.295591:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827971.295598:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564214400:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787827971.295620:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827971.295629:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827971.295635:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827971.295642:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564214528:192.168.203.139@tcp:38:kworker.0 00000100:00080000:2.0:1787827971.295653:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2911 00000100:00080000:2.0:1787827971.295665:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827971.295675:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:1.0:1787827971.295685:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:2.0:1787827971.295688:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787827971.295705:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827971/real 1787827971] req@ffff976889cfa940 x1874670564214528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827971.295730:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564214784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827971.295743:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564214528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827971.295759:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827971.295767:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827971.295768:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827971.295777:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827971.295783:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00080000:2.0:1787827971.295790:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2911 00000100:00100000:1.0:1787827971.295797:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564214528:192.168.203.139@tcp:38:kworker.0 00000100:00080000:2.0:1787827971.295801:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827971.295813:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827971.295818:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564214656:192.168.203.139@tcp:38:kworker.0 00000008:00100000:2.0:1787827971.295867:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827971.295893:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564214912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827971.295904:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827971.295942:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827971/real 1787827971] req@ffff976889cfbc00 x1874670564214656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827971.295980:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564214656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827971.295995:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827971.296003:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827971.296011:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564214656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827971.296050:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564214784:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827971.296080:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827971.296098:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827971/real 1787827971] req@ffff976889cf8000 x1874670564214784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827971.296119:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564214784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827971.296140:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827971.296145:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827971.296149:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564214784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827971.296156:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564214912:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827971.296174:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827971.296186:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827971/real 1787827971] req@ffff976889cfb0c0 x1874670564214912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827971.296199:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564214912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827971.296209:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827971.296213:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827971.296216:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564214912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827971.550948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827971.550974:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827971.551014:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827972.318956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827972.574993:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827972.575032:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827972.575087:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827973.342942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827973.598968:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827973.598973:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827973.599006:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827974.366945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827974.622935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827974.622941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827974.623025:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827975.390926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827975.646906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827975.646931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827975.646979:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827976.414925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827976.414943:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827976.414951:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827976.414956:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827976.414966:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2916 00000100:00080000:3.0:1787827976.414979:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827976.414998:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827976.415049:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564215040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827976.415075:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827976.415078:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827976.415080:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827976.415083:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2916 00000100:00080000:3.0:1787827976.415086:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827976.415090:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827976.415102:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827976.415118:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564215168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827976.415119:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564215040:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787827976.415135:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827976.415139:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827976.415140:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827976.415149:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2916 00000100:00080000:3.0:1787827976.415162:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827976.415165:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827976.415170:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827976.415180:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564215296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827976.415189:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827976.415191:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827976.415193:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827976.415196:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2916 00000100:00080000:3.0:1787827976.415201:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827976.415204:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827976.415208:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827976.415218:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564215424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827976.415241:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827976.415245:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827976.415246:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827976.415250:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2916 00000100:00080000:3.0:1787827976.415254:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827976.415257:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827976.415263:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827976.415274:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564215552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827976.415277:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564215168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827976.415315:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564215296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827976.415336:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564215424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827976.415353:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564215552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827976.670921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827976.670954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827976.671031:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827977.439072:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827977.694950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827977.694964:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827977.694966:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827978.463082:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827978.719053:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827978.719055:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827978.719126:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827979.487247:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:2.0:1787827979.487460:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:2.0:1787827979.487560:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787827979.487590:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787827979.487674:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787827979.487697:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787827979.487712:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787827979.487730:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:1.0:1787827979.487782:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827976/real 1787827979] req@ffff97688cb5c3c0 x1874670564215552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827979.487902:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564215552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827979.487937:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827979.487947:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827979.487956:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564215552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827979.487970:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827976/real 1787827979] req@ffff97688cb5cf00 x1874670564215168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827979.487995:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564215168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827979.488011:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827979.488018:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827979.488024:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564215168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827979.488050:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827976/real 1787827979] req@ffff97688cb5c780 x1874670564215296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827979.488070:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564215296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827979.488087:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827979.488094:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827979.488103:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564215296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827979.488115:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827976/real 1787827979] req@ffff97688cb5e1c0 x1874670564215424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827979.488136:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564215424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827979.488158:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827979.488164:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827979.488169:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564215424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827979.488180:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827976/real 1787827979] req@ffff97688cb5d2c0 x1874670564215040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827979.488199:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564215040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827979.488210:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827979.488215:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827979.488220:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564215040:192.168.203.139@tcp:250:kworker.0 00010000:00010000:3.0:1787827979.743022:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827979.743027:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827979.743029:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827980.510994:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827980.766987:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827980.767026:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827980.767080:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827981.534989:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827981.535006:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827981.535014:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827981.535032:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2921 00000100:00080000:3.0:1787827981.535051:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827981.535057:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827981.535065:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827981.535145:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564215680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827981.535193:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827981.535199:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827981.535202:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827981.535212:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2921 00000100:00080000:3.0:1787827981.535219:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827981.535229:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827981.535262:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827981.535290:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564215808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827981.535327:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827981.535333:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827981.535336:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827981.535346:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2921 00000100:00080000:3.0:1787827981.535353:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827981.535364:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787827981.535398:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564215680:192.168.203.139@tcp:250:kworker.0 00000008:00100000:3.0:1787827981.535401:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827981.535426:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564215936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827981.535440:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827981.535445:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827981.535448:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827981.535454:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2921 00000100:00080000:3.0:1787827981.535459:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827981.535464:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827981.535472:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787827981.535486:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564216064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827981.535499:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827981.535503:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827981.535506:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827981.535512:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2921 00000100:00080000:3.0:1787827981.535518:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827981.535522:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827981.535528:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827981.535540:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827981.535546:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564216192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827981.535622:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827981/real 1787827981] req@ffff97688cb5f840 x1874670564215680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827981.535659:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564215680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827981.535693:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827981.535702:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827981.535710:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564215680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827981.535727:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564215808:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827981.535775:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827981.535802:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827981/real 1787827981] req@ffff97688cb5cb40 x1874670564215808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827981.535898:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564215808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827981.535918:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827981.535925:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827981.535932:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564215808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827981.535988:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564215936:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827981.536045:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827981.536069:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827981/real 1787827981] req@ffff97688cb5c000 x1874670564215936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827981.536093:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564215936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827981.536122:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827981.536129:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827981.536135:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564215936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827981.536146:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564216064:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827981.536185:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827981.536207:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827981/real 1787827981] req@ffff97688cb5d680 x1874670564216064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827981.536228:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564216064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827981.536242:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827981.536248:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827981.536254:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564216064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827981.536264:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564216192:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827981.536310:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827981.536330:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827981/real 1787827981] req@ffff97688cb5f0c0 x1874670564216192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827981.536352:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564216192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827981.536407:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827981.536414:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827981.536422:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564216192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827981.790950:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827981.791006:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827981.791036:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827982.559070:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827982.814976:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827982.815044:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827982.815056:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827983.582962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827983.838961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827983.838998:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827983.839028:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787827984.607018:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827984.862931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827984.862931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827984.863069:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827985.630904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827985.886914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827985.886916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827985.886941:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827986.654917:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827986.654927:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827986.654931:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827986.654941:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2926 00000100:00080000:0.0:1787827986.654951:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827986.654958:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827986.654997:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564216320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827986.655022:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827986.655026:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827986.655027:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827986.655031:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2926 00000100:00080000:0.0:1787827986.655034:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827986.655037:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827986.655046:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827986.655058:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564216448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827986.655074:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827986.655077:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827986.655079:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827986.655082:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2926 00000100:00080000:0.0:1787827986.655085:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827986.655088:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827986.655093:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827986.655100:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564216576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827986.655110:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827986.655111:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564216320:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787827986.655115:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827986.655119:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827986.655125:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2926 00000100:00080000:0.0:1787827986.655128:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827986.655130:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827986.655134:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827986.655145:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564216704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827986.655153:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827986.655156:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827986.655158:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827986.655162:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2926 00000100:00080000:0.0:1787827986.655176:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827986.655180:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827986.655184:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827986.655188:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787827986.655199:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564216832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827986.655233:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827986/real 1787827986] req@ffff976882aeed00 x1874670564216320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827986.655236:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827986.655257:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564216320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827986.655269:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827986.655273:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827986.655277:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564216320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827986.655284:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564216448:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827986.655301:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827986.655310:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827986/real 1787827986] req@ffff976882aec3c0 x1874670564216448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827986.655320:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564216448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827986.655327:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827986.655330:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827986.655350:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564216448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827986.655393:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564216576:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827986.655414:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827986.655424:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827986/real 1787827986] req@ffff976882aee580 x1874670564216576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827986.655454:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564216576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827986.655465:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827986.655468:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827986.655473:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564216576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827986.655480:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564216704:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827986.655502:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827986.655513:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827986/real 1787827986] req@ffff976882aed680 x1874670564216704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827986.655527:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564216704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827986.655536:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827986.655539:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827986.655543:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564216704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827986.655549:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564216832:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827986.655567:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827986.655599:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827986/real 1787827986] req@ffff976882aee940 x1874670564216832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827986.655610:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564216832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827986.655617:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827986.655620:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827986.655622:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564216832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827986.910893:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827986.910907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827986.910951:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827987.678889:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827987.934937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827987.934947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827987.934951:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827988.702903:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827988.958901:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827988.958907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827988.958954:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827989.726984:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827989.982954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787827989.982963:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787827989.982995:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827990.750984:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787827991.006931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827991.006970:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787827991.007103:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827991.774983:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827991.774989:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787827991.774998:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827991.775006:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827991.775019:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2931 00000100:00080000:0.0:1787827991.775040:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827991.775052:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787827991.775103:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564216960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827991.775145:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827991.775151:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827991.775160:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827991.775168:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2931 00000100:00080000:0.0:1787827991.775176:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827991.775183:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827991.775198:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827991.775217:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564216960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827991.775221:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564217088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827991.775238:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827991.775243:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827991.775246:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827991.775251:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2931 00000100:00080000:0.0:1787827991.775257:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827991.775262:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827991.775269:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827991.775285:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564217216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827991.775296:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787827991.775323:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827991.775328:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827991.775330:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827991.775336:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2931 00000100:00080000:0.0:1787827991.775341:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827991.775347:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787827991.775354:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787827991.775371:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564217344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827991.775374:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827991/real 1787827991] req@ffff976882aef0c0 x1874670564216960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827991.775399:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827991.775403:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564216960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827991.775404:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827991.775407:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827991.775413:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2931 00000100:00080000:0.0:1787827991.775418:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827991.775421:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827991.775423:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:1.0:1787827991.775428:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000008:00100000:0.0:1787827991.775430:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827991.775435:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564216960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787827991.775443:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564217472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827991.775468:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564217088:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827991.775513:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827991.775528:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827991/real 1787827991] req@ffff976882aee940 x1874670564217088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827991.775544:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564217088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827991.775555:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827991.775559:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827991.775562:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564217088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827991.775568:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564217216:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827991.775586:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827991.775595:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827991/real 1787827991] req@ffff976882aed680 x1874670564217216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827991.775603:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564217216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827991.775608:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827991.775610:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827991.775612:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564217216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827991.775616:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564217344:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827991.775626:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827991.775633:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827991/real 1787827991] req@ffff976882aee580 x1874670564217344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827991.775640:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564217344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827991.775646:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827991.775648:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827991.775650:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564217344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827991.775653:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564217472:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827991.775663:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827991.775669:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827991/real 1787827991] req@ffff976882aec3c0 x1874670564217472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827991.775675:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564217472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827991.775680:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827991.775682:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827991.775684:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564217472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787827992.030955:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787827992.031025:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787827992.031067:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827992.798943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827993.054938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787827993.055074:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787827993.055127:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827993.822914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827994.078935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827994.078992:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827994.079118:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827994.846910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827995.102958:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827995.103005:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827995.103105:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827995.871035:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827996.126971:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827996.127022:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827996.127098:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827996.894997:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827996.895013:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827996.895021:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827996.895040:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2936 00000100:00100000:2.0:1787827996.895060:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787827996.895063:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827996.895084:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787827996.895165:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564217600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827996.895218:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827996.895224:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827996.895226:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827996.895233:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2936 00000100:00080000:3.0:1787827996.895239:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827996.895246:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827996.895287:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827996.895313:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564217600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787827996.895317:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564217728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827996.895339:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827996.895344:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827996.895347:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827996.895352:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2936 00000100:00080000:3.0:1787827996.895358:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827996.895363:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827996.895370:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827996.895389:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564217856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827996.895401:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827996.895405:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827996.895408:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827996.895413:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2936 00000100:00080000:3.0:1787827996.895419:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827996.895424:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827996.895430:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1787827996.895452:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787827996.895453:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564217984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827996.895471:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827996.895477:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827996.895480:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827996.895485:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2936 00000100:00080000:3.0:1787827996.895490:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827996.895495:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787827996.895500:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827996.895514:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564218112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827996.895542:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827996/real 1787827996] req@ffff97688cb5de00 x1874670564217600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827996.895585:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564217600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827996.895601:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827996.895607:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827996.895612:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564217600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787827996.895656:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564217728:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827996.895688:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827996.895705:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827996/real 1787827996] req@ffff97688cb5f0c0 x1874670564217728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827996.895721:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564217728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827996.895733:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827996.895737:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827996.895741:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564217728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827996.895748:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564217856:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787827996.895770:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827996.895781:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827996/real 1787827996] req@ffff97688cb5d680 x1874670564217856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827996.895795:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564217856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827996.895863:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827996.895871:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827996.895879:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564217856:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787827996.895892:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564217984:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827996.895933:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827996.895950:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827996/real 1787827996] req@ffff97688cb5c000 x1874670564217984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827996.895965:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564217984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827996.895976:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827996.895980:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827996.895999:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564217984:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827996.896007:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564218112:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787827996.896031:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787827996.896043:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827996/real 1787827996] req@ffff97688cb5cb40 x1874670564218112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827996.896058:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564218112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827996.896067:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827996.896071:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787827996.896076:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564218112:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787827997.150974:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787827997.151023:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787827997.151162:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827997.919026:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787827998.175056:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827998.175063:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827998.175106:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787827998.942990:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787827999.198919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787827999.198989:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787827999.199103:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827999.966961:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828000.222953:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828000.223024:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828000.223125:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828000.990939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828001.246906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828001.246916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828001.247005:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828002.014889:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828002.014897:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828002.014908:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828002.014917:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2941 00000100:00080000:3.0:1787828002.014926:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828002.014932:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828002.014957:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787828002.014970:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564218240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828002.014996:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828002.015001:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828002.015004:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828002.015008:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2941 00000100:00080000:3.0:1787828002.015010:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828002.015013:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828002.015021:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828002.015031:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564218368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828002.015039:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828002.015042:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564218240:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787828002.015043:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828002.015047:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828002.015053:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2941 00000100:00080000:3.0:1787828002.015056:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828002.015058:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828002.015060:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828002.015070:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564218496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828002.015076:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828002.015078:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828002.015080:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828002.015083:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2941 00000100:00080000:3.0:1787828002.015086:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828002.015088:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828002.015092:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828002.015100:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564218624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828002.015107:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787828002.015114:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828002.015117:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828002.015118:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828002.015122:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2941 00000100:00080000:3.0:1787828002.015125:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828002.015128:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828002.015131:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828002.015139:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564218752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828002.015157:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828002/real 1787828002] req@ffff97688cb5f840 x1874670564218240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828002.015172:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564218240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828002.015181:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828002.015184:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828002.015187:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564218240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828002.015213:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564218368:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828002.015248:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828002.015272:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828002/real 1787828002] req@ffff97688cb5d2c0 x1874670564218368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828002.015285:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564218368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828002.015294:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828002.015297:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828002.015302:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564218368:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828002.015308:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564218496:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828002.015324:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828002.015331:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828002/real 1787828002] req@ffff97688cb5e1c0 x1874670564218496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828002.015339:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564218496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828002.015346:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828002.015348:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828002.015350:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564218496:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828002.015353:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564218624:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828002.015374:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828002.015380:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828002/real 1787828002] req@ffff97688cb5c780 x1874670564218624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828002.015387:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564218624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828002.015392:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828002.015394:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828002.015396:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564218624:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828002.015399:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564218752:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828002.015409:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828002.015415:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828002/real 1787828002] req@ffff97688cb5cf00 x1874670564218752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828002.015423:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564218752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828002.015429:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828002.015432:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828002.015435:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564218752:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828002.270907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828002.270980:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828002.270992:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828003.038992:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828003.294987:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828003.295005:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828003.295180:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828004.062877:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828004.318906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828004.318906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828004.318985:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828005.087045:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828005.342916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828005.342952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828005.343074:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828006.110962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828006.366970:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828006.366980:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828006.367025:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828007.135067:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828007.135075:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828007.135078:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828007.135088:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828007.135104:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2947 00000100:00080000:3.0:1787828007.135118:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828007.135130:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828007.135177:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564218880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828007.135266:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828007.135273:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828007.135278:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828007.135287:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2947 00000100:00080000:3.0:1787828007.135296:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828007.135305:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828007.135323:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828007.135333:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564218880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828007.135361:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564219008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828007.135380:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828007.135385:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828007.135389:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828007.135396:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2947 00000100:00080000:3.0:1787828007.135414:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828007.135420:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:2.0:1787828007.135429:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:3.0:1787828007.135431:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828007.135454:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564219136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828007.135466:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828007.135471:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828007.135473:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828007.135480:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2947 00000100:00080000:3.0:1787828007.135485:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828007.135492:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828007/real 1787828007] req@ffff97688cb5c3c0 x1874670564218880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828007.135515:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564218880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828007.135529:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828007.135534:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828007.135539:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564218880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828007.135567:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564219008:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828007.135595:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828007.135613:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828007/real 1787828007] req@ffff97688cb5cf00 x1874670564219008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828007.135631:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564219008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828007.135641:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828007.135645:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828007.135649:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564219008:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828007.135655:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564219136:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828007.135677:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828007.135690:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828007/real 1787828007] req@ffff97688cb5c780 x1874670564219136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828007.135704:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564219136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828007.135712:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828007.135716:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828007.135720:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564219136:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787828007.135792:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828007.135802:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828007.135857:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564219264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828007.135880:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828007.135885:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828007.135889:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828007.135895:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2947 00000100:00080000:3.0:1787828007.135902:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828007.135908:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828007.135915:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828007.135937:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564219264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828007.135944:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564219392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828007.135987:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828007.136006:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828007/real 1787828007] req@ffff97688cb5e1c0 x1874670564219264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828007.136043:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564219264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828007.136082:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828007.136086:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828007.136097:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564219264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828007.136131:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564219392:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828007.136161:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828007.136183:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828007/real 1787828007] req@ffff97688cb5d2c0 x1874670564219392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828007.136265:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564219392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828007.136283:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828007.136290:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828007.136297:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564219392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828007.390938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828007.390962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828007.391049:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828008.158915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828008.414953:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828008.414962:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828008.414987:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828009.182988:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828009.438974:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828009.438974:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828009.439174:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828010.206913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828010.462906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828010.462966:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828010.463124:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828011.230898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828011.486907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828011.487021:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828011.487113:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828012.255040:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828012.255056:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1787828012.255061:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828012.255064:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828012.255090:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2952 00000100:00080000:3.0:1787828012.255119:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828012.255131:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828012.255230:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564219520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828012.255275:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828012.255281:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828012.255284:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828012.255289:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2952 00000100:00080000:3.0:1787828012.255295:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828012.255303:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828012.255319:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828012.255343:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564219648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828012.255365:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564219520:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787828012.255381:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828012.255389:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828012.255391:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828012.255397:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2952 00000100:00080000:3.0:1787828012.255402:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828012.255407:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828012.255414:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828012.255434:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564219776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828012.255449:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828012.255454:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828012.255457:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828012.255466:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2952 00000800:00000100:2.0:1787828012.255471:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787828012.255476:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828012.255485:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828012.255494:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828012.255509:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564219904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828012.255523:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828012.255540:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828012.255543:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828012.255550:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2952 00000100:00080000:3.0:1787828012.255555:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828012.255564:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828012.255568:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828012/real 1787828012] req@ffff97688cb5f840 x1874670564219520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828012.255570:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828012.255592:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564220032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828012.255665:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564219520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828012.255685:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828012.255691:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828012.255697:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564219520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828012.255753:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564219648:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828012.255792:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828012.255827:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828012/real 1787828012] req@ffff97688cb5cb40 x1874670564219648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828012.255874:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564219648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828012.255886:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828012.255890:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828012.255893:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564219648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828012.255901:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564219776:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828012.255945:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828012.255960:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828012/real 1787828012] req@ffff97688cb5c000 x1874670564219776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828012.255975:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564219776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828012.255986:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828012.256006:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828012.256010:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564219776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828012.256017:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564219904:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828012.256038:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828012.256048:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828012/real 1787828012] req@ffff97688cb5d680 x1874670564219904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828012.256079:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564219904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828012.256090:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828012.256093:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828012.256097:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564219904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828012.256104:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564220032:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828012.256126:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828012.256137:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828012/real 1787828012] req@ffff97688cb5f0c0 x1874670564220032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828012.256152:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564220032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828012.256210:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828012.256216:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828012.256221:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564220032:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828012.510951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828012.511005:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828012.511180:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828013.278940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828013.535067:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828013.535072:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828013.535116:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828014.303002:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828014.558999:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828014.559047:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828014.559072:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828015.326954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828015.583044:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828015.583044:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828015.583209:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828016.350956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828016.606926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828016.606940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828016.607004:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828017.374962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828017.374985:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828017.374997:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828017.375004:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828017.375017:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2957 00000100:00080000:0.0:1787828017.375030:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828017.375038:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828017.375083:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564220160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828017.375143:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828017.375150:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828017.375171:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828017.375182:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2957 00000100:00080000:0.0:1787828017.375190:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828017.375199:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828017.375216:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828017.375224:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564220160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828017.375246:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564220288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828017.375269:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828017.375274:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828017.375278:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828017.375284:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2957 00000100:00080000:0.0:1787828017.375291:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828017.375298:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828017.375305:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828017.375321:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564220416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828017.375336:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828017.375341:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828017.375343:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828017.375348:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2957 00000100:00080000:0.0:1787828017.375354:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828017.375365:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828017.375376:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828017.375386:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828017.375407:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564220544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828017.375420:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828017.375424:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828017.375427:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828017.375450:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2957 00000100:00080000:0.0:1787828017.375457:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828017.375464:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828017/real 1787828017] req@ffff976882aeed00 x1874670564220160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828017.375466:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828017.375481:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828017.375510:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564220672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828017.375522:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564220160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828017.375584:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828017.375593:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828017.375603:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564220160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828017.375648:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564220288:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828017.375695:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828017.375723:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828017/real 1787828017] req@ffff976882aec3c0 x1874670564220288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828017.375749:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564220288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828017.375768:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828017.375773:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828017.375778:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564220288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828017.375787:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564220416:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828017.375819:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828017.375871:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828017/real 1787828017] req@ffff976882aee580 x1874670564220416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828017.375889:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564220416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828017.375900:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828017.375904:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828017.375908:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564220416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828017.375917:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564220544:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828017.375943:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828017.375955:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828017/real 1787828017] req@ffff976882aed680 x1874670564220544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828017.375970:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564220544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828017.375980:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828017.375984:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828017.375987:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564220544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828017.375994:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564220672:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828017.376013:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828017.376024:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828017/real 1787828017] req@ffff976882aee940 x1874670564220672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828017.376037:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564220672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828017.376056:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828017.376060:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828017.376065:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564220672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828017.630951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828017.631003:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828017.631057:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828018.398997:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828018.654962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828018.654998:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828018.655171:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828019.422909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828019.678941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828019.678942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828019.679003:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828020.446945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828020.702991:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828020.702992:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828020.703077:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828021.470968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828021.726979:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828021.726997:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828021.727111:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828022.494957:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828022.495041:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828022.495056:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828022.495063:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828022.495078:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2962 00000100:00080000:3.0:1787828022.495115:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828022.495138:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828022.495196:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564220800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828022.495241:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828022.495246:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828022.495249:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828022.495255:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2962 00000100:00080000:3.0:1787828022.495259:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828022.495264:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828022.495288:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828022.495308:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564220928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828022.495336:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828022.495343:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564220800:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787828022.495353:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828022.495359:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828022.495371:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2962 00000100:00080000:3.0:1787828022.495376:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828022.495381:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828022.495389:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828022.495405:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564221056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828022.495418:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828022.495423:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828022.495426:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828022.495431:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2962 00000100:00080000:3.0:1787828022.495447:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828022.495452:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828022.495459:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828022.495473:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564221184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828022.495485:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787828022.495499:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828022.495504:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828022.495507:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828022.495514:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2962 00000100:00080000:3.0:1787828022.495520:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828022.495526:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828022.495532:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828022.495547:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564221312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828022.495577:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828022/real 1787828022] req@ffff97688cb5de00 x1874670564220800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828022.495617:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564220800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828022.495633:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828022.495639:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828022.495644:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564220800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828022.495698:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564220928:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828022.495730:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828022.495749:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828022/real 1787828022] req@ffff97688cb5f0c0 x1874670564220928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828022.495765:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564220928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828022.495783:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828022.495789:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828022.495797:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564220928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828022.495808:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564221056:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828022.495958:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828022.495984:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828022/real 1787828022] req@ffff97688cb5d680 x1874670564221056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828022.496000:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564221056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828022.496012:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828022.496016:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828022.496030:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564221056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828022.496038:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564221184:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828022.496063:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828022.496074:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828022/real 1787828022] req@ffff97688cb5c000 x1874670564221184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828022.496117:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564221184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828022.496129:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828022.496145:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828022.496149:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564221184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828022.496157:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564221312:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828022.496186:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828022.496199:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828022/real 1787828022] req@ffff97688cb5cb40 x1874670564221312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828022.496213:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564221312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828022.496224:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828022.496227:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828022.496231:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564221312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828022.750943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828022.750975:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828022.751143:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828023.518970:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828023.774963:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828023.774989:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828023.775016:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828024.542889:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828024.798941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828024.798957:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828024.798987:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828025.566926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828025.822891:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828025.822928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828025.822938:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828026.590898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828026.846883:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828026.846896:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828026.846974:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828027.614964:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828027.614980:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828027.614988:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828027.614993:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828027.615005:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2967 00000100:00080000:3.0:1787828027.615023:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828027.615036:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828027.615112:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564221440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828027.615163:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828027.615170:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828027.615174:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828027.615184:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2967 00000100:00080000:3.0:1787828027.615192:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828027.615202:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564221440:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787828027.615205:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828027.615228:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828027.615260:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564221568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828027.615282:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828027.615305:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:2.0:1787828027.615313:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787828027.615314:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828027.615328:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2967 00000100:00080000:3.0:1787828027.615337:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828027.615346:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828027.615357:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828027.615367:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828027/real 1787828027] req@ffff97688cb5f840 x1874670564221440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828027.615385:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564221696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828027.615397:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564221440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828027.615432:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828027.615437:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828027.615442:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828027.615447:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:3.0:1787828027.615451:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828027.615457:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564221440:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787828027.615468:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2967 00000100:00080000:3.0:1787828027.615486:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828027.615494:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828027.615506:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828027.615508:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564221568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828027.615538:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564221824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828027.615569:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828027.615575:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828027.615578:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828027.615591:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2967 00000800:00000100:2.0:1787828027.615593:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787828027.615603:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828027.615614:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828027.615631:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828027.615642:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828027/real 1787828027] req@ffff97688cb5d2c0 x1874670564221568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828027.615657:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564221952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828027.615706:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564221568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828027.615721:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828027.615726:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828027.615730:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564221568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828027.615738:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564221696:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828027.615769:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828027.615784:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828027/real 1787828027] req@ffff97688cb5e1c0 x1874670564221696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828027.615816:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564221696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828027.615879:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828027.615884:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828027.615888:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564221696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828027.615913:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564221824:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828027.615942:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828027.615962:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828027/real 1787828027] req@ffff97688cb5c780 x1874670564221824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828027.615978:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564221824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828027.615988:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828027.615991:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828027.615995:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564221824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828027.616001:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564221952:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828027.616026:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828027.616037:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828027/real 1787828027] req@ffff97688cb5cf00 x1874670564221952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828027.616077:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564221952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828027.616106:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828027.616110:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828027.616114:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564221952:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828027.870936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828027.871045:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828027.871107:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828028.638921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828028.895008:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828028.895020:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828028.895081:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828029.662962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828029.919026:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828029.919032:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828029.919111:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828030.686934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828030.942978:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828030.943042:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828030.943052:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828031.710895:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828031.966906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828031.966926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828031.967002:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828032.734920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828032.734960:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828032.734971:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828032.734977:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828032.734989:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2972 00000100:00080000:3.0:1787828032.735002:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828032.735011:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828032.735087:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564222080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828032.735133:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828032.735139:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828032.735141:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828032.735147:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2972 00000100:00080000:3.0:1787828032.735153:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828032.735159:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828032.735172:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828032.735192:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564222080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828032.735195:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564222208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828032.735218:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828032.735223:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828032.735227:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828032.735232:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2972 00000100:00080000:3.0:1787828032.735237:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828032.735242:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828032.735249:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828032.735263:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564222336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828032.735276:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828032.735280:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787828032.735287:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787828032.735288:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828032.735302:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2972 00000100:00080000:3.0:1787828032.735309:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828032.735315:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828032.735322:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828032.735338:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564222464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828032.735350:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828032.735353:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828032.735356:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828032.735368:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2972 00000100:00080000:3.0:1787828032.735374:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828032.735385:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828032.735386:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828032/real 1787828032] req@ffff97688cb5c3c0 x1874670564222080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828032.735391:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828032.735421:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564222592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828032.735431:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564222080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828032.735459:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828032.735465:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828032.735472:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564222080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828032.735505:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564222208:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828032.735546:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828032.735573:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828032/real 1787828032] req@ffff97688cb5cf00 x1874670564222208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828032.735600:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564222208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828032.735619:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828032.735626:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828032.735632:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564222208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828032.735640:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564222336:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828032.735681:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828032.735705:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828032/real 1787828032] req@ffff97688cb5c780 x1874670564222336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828032.735731:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564222336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828032.735749:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828032.735767:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828032.735771:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564222336:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828032.735780:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564222464:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828032.735811:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828032.735855:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828032/real 1787828032] req@ffff97688cb5e1c0 x1874670564222464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828032.735870:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564222464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828032.735882:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828032.735886:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828032.735890:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564222464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828032.735897:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564222592:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828032.735920:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828032.735932:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828032/real 1787828032] req@ffff97688cb5d2c0 x1874670564222592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828032.735946:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564222592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828032.735956:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828032.735959:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828032.735963:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564222592:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828032.990975:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828032.990975:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828032.991074:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828033.758981:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787828034.014999:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828034.015031:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828034.015032:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828034.782946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828035.038968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828035.039043:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828035.039051:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828035.806955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828036.062912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828036.062942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828036.062962:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828036.830951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828037.086962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828037.086963:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828037.087025:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828037.855021:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828037.855027:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828037.855036:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828037.855046:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828037.855062:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2977 00000100:00080000:3.0:1787828037.855090:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828037.855103:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828037.855157:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564222720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828037.855199:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828037.855206:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828037.855211:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828037.855221:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2977 00000100:00080000:3.0:1787828037.855230:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828037.855249:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828037.855261:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564222720:192.168.203.139@tcp:250:kworker.0 00000008:00100000:3.0:1787828037.855266:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828037.855311:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564222848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828037.855328:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828037.855334:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828037.855337:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828037.855342:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2977 00000100:00080000:3.0:1787828037.855352:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828037.855357:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828037.855364:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828037.855382:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828037.855383:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564222976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828037.855411:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828037.855416:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828037.855418:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828037.855425:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2977 00000100:00080000:3.0:1787828037.855430:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828037.855435:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828037.855442:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828037.855456:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564223104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828037.855468:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828037.855472:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828037.855474:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828037.855479:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2977 00000100:00080000:3.0:1787828037.855483:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828037.855494:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828037.855496:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828037/real 1787828037] req@ffff97688cb5f840 x1874670564222720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828037.855508:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828037.855541:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564223232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828037.855554:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564222720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828037.855590:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828037.855596:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828037.855603:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564222720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828037.855642:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564222848:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828037.855678:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828037.855696:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828037/real 1787828037] req@ffff97688cb5cb40 x1874670564222848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828037.855712:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564222848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828037.855723:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828037.855727:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828037.855731:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564222848:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828037.855739:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564222976:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828037.855759:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828037.855770:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828037/real 1787828037] req@ffff97688cb5c000 x1874670564222976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828037.855792:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564222976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828037.855802:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828037.855806:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828037.855809:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564222976:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828037.855816:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564223104:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828037.855879:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828037.855894:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828037/real 1787828037] req@ffff97688cb5d680 x1874670564223104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828037.855909:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564223104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828037.855919:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828037.855933:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828037.855936:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564223104:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828037.855943:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564223232:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828037.855962:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828037.855973:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828037/real 1787828037] req@ffff97688cb5f0c0 x1874670564223232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828037.856013:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564223232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828037.856037:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828037.856041:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828037.856045:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564223232:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828038.110955:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828038.111016:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828038.111059:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828038.878933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828039.134988:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828039.135001:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828039.135046:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828039.903039:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828040.158905:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828040.158933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828040.159002:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828040.926952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828041.182954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828041.183035:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828041.183080:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828041.951003:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828042.206983:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828042.207021:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828042.207067:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00080000:3.0:1787828042.975012:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828042.975014:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828042.975035:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828042.975041:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828042.975056:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2982 00000100:00080000:3.0:1787828042.975070:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828042.975078:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828042.975131:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564223360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828042.975172:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828042.975177:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828042.975180:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828042.975186:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2982 00000100:00080000:3.0:1787828042.975200:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828042.975206:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828042.975218:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828042.975236:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564223488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828042.975259:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828042.975261:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564223360:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787828042.975281:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828042.975283:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828042.975290:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2982 00000100:00080000:3.0:1787828042.975296:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828042.975401:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828042.975418:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828042.975526:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564223488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828042.975611:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564223616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828042.975636:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828042.975642:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828042.975647:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828042.975661:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2982 00000100:00100000:2.0:1787828042.975674:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564223616:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787828042.975686:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828042.975700:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828042.975713:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828042.976853:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564223744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828042.976895:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828042.976900:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828042.976904:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828042.976930:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2982 00000100:00080000:3.0:1787828042.976958:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828042.976967:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828042.976978:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828042.977004:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564223872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828042.977066:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564223744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828042.977147:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564223872:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828043.230912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828043.230937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828043.231055:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828043.999006:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828044.255063:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828044.255063:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828044.255079:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828045.022959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828045.278945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828045.278985:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828045.279026:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828045.982886:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828045.982898:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787828045.982902:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828045.982906:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787828045.982908:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828045.982911:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787828045.982914:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828045.982916:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787828045.982918:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828045.982937:0:42426:0:(pinger.c:247:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1787828046.046983:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:0.0:1787828046.047197:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:0.0:1787828046.047250:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828046.047270:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828046.047312:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828046.047333:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828046.047343:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828046.047352:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:2.0:1787828046.047361:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828042/real 1787828046] req@ffff97688cb5c000 x1874670564223744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828046.047391:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564223744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828046.047413:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828046.047418:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828046.047423:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564223744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828046.047432:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828042/real 1787828046] req@ffff97688cb5cb40 x1874670564223872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828046.047446:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564223872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828046.047454:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828046.047458:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828046.047462:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564223872:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828046.047468:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828042/real 1787828046] req@ffff97688cb5d680 x1874670564223616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828046.047480:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564223616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828046.047489:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828046.047494:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828046.047497:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564223616:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828046.047509:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828042/real 1787828046] req@ffff97688cb5f0c0 x1874670564223488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828046.047523:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564223488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828046.047532:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828046.047537:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828046.047541:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564223488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828046.047549:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828042/real 1787828046] req@ffff97688cb5de00 x1874670564223360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828046.047565:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564223360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828046.047573:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828046.047578:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828046.047582:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564223360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828046.303007:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828046.303007:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828046.303108:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828047.006939:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828047.006955:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828047.006963:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828047.006982:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2987 00000100:00080000:3.0:1787828047.006998:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828047.007011:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828047.007064:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564224000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828047.007104:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828047.007108:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828047.007111:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828047.007117:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2987 00000100:00080000:3.0:1787828047.007122:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828047.007127:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828047.007140:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828047.007157:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564224128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828047.007174:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828047.007179:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828047.007184:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828047.007195:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564224000:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787828047.007196:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2987 00000100:00080000:3.0:1787828047.007211:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828047.007222:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828047.007228:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828047.007246:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564224256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828047.007267:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828047.007272:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828047.007274:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828047.007281:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2987 00000100:00080000:3.0:1787828047.007310:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828047.007315:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828047.007321:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828047.007334:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564224384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828047.007351:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787828047.007359:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828047.007367:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828047.007370:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828047.007377:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2987 00000100:00080000:3.0:1787828047.007382:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828047.007389:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828047.007396:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828047.007413:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564224512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828047.007447:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828047/real 1787828047] req@ffff97688cb5f840 x1874670564224000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828047.007477:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564224000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828047.007503:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828047.007510:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828047.007516:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564224000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828047.007562:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564224128:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828047.007600:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828047.007617:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828047/real 1787828047] req@ffff97688cb5d2c0 x1874670564224128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828047.007633:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564224128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828047.007650:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828047.007658:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828047.007664:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564224128:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828047.007679:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564224256:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828047.007742:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828047.007764:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828047/real 1787828047] req@ffff97688cb5e1c0 x1874670564224256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828047.007789:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564224256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828047.007807:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828047.007815:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828047.007876:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564224256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828047.007890:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564224384:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828047.007957:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828047.007976:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828047/real 1787828047] req@ffff97688cb5c780 x1874670564224384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828047.007993:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564224384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828047.008004:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828047.008009:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828047.008013:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564224384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828047.008022:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564224512:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828047.008047:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828047.008058:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828047/real 1787828047] req@ffff97688cb5cf00 x1874670564224512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828047.008072:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564224512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828047.008082:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828047.008085:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828047.008089:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564224512:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828047.070976:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828047.326992:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828047.327015:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828047.327051:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828048.094984:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828048.350958:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828048.351011:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828048.351016:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828049.119000:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828049.375024:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828049.375051:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828049.375063:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828050.142990:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828050.398996:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828050.398996:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828050.399041:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828051.166965:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828051.422995:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828051.422999:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828051.423079:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828052.190967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828052.191005:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828052.191016:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828052.191023:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828052.191038:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2992 00000100:00080000:1.0:1787828052.191054:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828052.191066:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828052.191126:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564224640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828052.191169:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828052.191174:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828052.191177:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828052.191182:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2992 00000100:00080000:1.0:1787828052.191187:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828052.191191:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828052.191205:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828052.191223:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564224768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828052.191238:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828052.191243:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828052.191245:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828052.191250:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2992 00000100:00080000:1.0:1787828052.191254:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828052.191260:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564224640:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787828052.191263:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828052.191276:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828052.191290:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564224896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828052.191304:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828052.191309:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828052.191312:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828052.191317:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2992 00000100:00080000:1.0:1787828052.191323:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828052.191328:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828052.191335:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828052.191349:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564225024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828052.191360:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828052.191365:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828052.191367:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828052.191372:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2992 00000800:00000100:2.0:1787828052.191381:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787828052.191383:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828052.191393:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828052.191400:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828052.191416:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564225152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828052.191472:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828052/real 1787828052] req@ffff976889efcb40 x1874670564224640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828052.191500:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564224640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828052.191517:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828052.191523:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828052.191530:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564224640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828052.191541:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564224768:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828052.191570:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828052.191585:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828052/real 1787828052] req@ffff976889efe1c0 x1874670564224768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828052.191601:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564224768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828052.191612:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828052.191616:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828052.191622:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564224768:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828052.191656:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564224896:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828052.191678:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828052.191690:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828052/real 1787828052] req@ffff976889efe580 x1874670564224896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828052.191704:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564224896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828052.191714:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828052.191718:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828052.191722:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564224896:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828052.191728:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564225024:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828052.191748:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828052.191759:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828052/real 1787828052] req@ffff976889eff480 x1874670564225024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828052.191773:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564225024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828052.191783:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828052.191786:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828052.191790:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564225024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828052.191796:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564225152:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828052.191814:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828052.191862:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828052/real 1787828052] req@ffff976889efc780 x1874670564225152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828052.191907:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564225152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828052.191920:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828052.191924:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828052.191929:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564225152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828052.446941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828052.446966:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828052.446985:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828053.214940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828053.470982:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828053.471004:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828053.471011:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828054.238976:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828054.494978:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828054.494993:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828054.495111:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828055.262992:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828055.518923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828055.518954:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828055.518972:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828056.286901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828056.542924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828056.542924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828056.542967:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828057.310959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828057.311092:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828057.311112:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828057.311122:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828057.311155:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 2997 00000100:00080000:0.0:1787828057.311177:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828057.311191:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828057.311266:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564225280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828057.311300:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828057.311305:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828057.311307:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828057.311313:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2997 00000100:00080000:0.0:1787828057.311320:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828057.311326:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828057.311360:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828057.311375:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564225280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828057.311394:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564225408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828057.311416:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828057.311423:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828057.311427:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828057.311433:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2997 00000100:00080000:0.0:1787828057.311440:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828057.311450:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828057.311463:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828057.311474:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828057.311484:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564225536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828057.311503:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828057.311509:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828057.311513:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828057.311522:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2997 00000100:00080000:0.0:1787828057.311528:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828057.311535:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828057/real 1787828057] req@ffff976882aef0c0 x1874670564225280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828057.311542:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828057.311562:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828057.311580:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564225664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828057.311607:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564225280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828057.311615:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828057.311640:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828057.311648:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828057.311653:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828057.311659:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 2997 00000100:00080000:2.0:1787828057.311681:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:0.0:1787828057.311688:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828057.311699:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564225280:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828057.311705:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828057.311718:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828057.311736:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564225792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828057.311765:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564225408:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828057.311815:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828057.311912:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828057/real 1787828057] req@ffff976882aee940 x1874670564225408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828057.311931:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564225408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828057.311953:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828057.311958:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828057.311965:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564225408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828057.311976:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564225536:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828057.312026:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828057.312045:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828057/real 1787828057] req@ffff976882aed680 x1874670564225536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828057.312059:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564225536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828057.312068:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828057.312071:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828057.312075:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564225536:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828057.312090:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564225664:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828057.312109:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828057.312119:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828057/real 1787828057] req@ffff976882aee580 x1874670564225664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828057.312131:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564225664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828057.312140:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828057.312143:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828057.312146:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564225664:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828057.312152:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564225792:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828057.312169:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828057.312198:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828057/real 1787828057] req@ffff976882aec3c0 x1874670564225792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828057.312216:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564225792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828057.312241:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828057.312246:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828057.312251:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564225792:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828057.566948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828057.566960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828057.567076:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828058.334989:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828058.590993:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828058.591007:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828058.591066:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828059.359010:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828059.615013:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828059.615055:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828059.615176:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828060.382983:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828060.638926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828060.638929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828060.639004:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828061.406986:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828061.662975:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828061.662998:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828061.663088:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828062.430959:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828062.430972:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1787828062.430975:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828062.430979:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828062.430997:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3002 00000100:00080000:0.0:1787828062.431010:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828062.431021:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828062.431072:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564225920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828062.431109:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828062.431125:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828062.431128:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828062.431134:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3002 00000100:00080000:0.0:1787828062.431141:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828062.431147:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828062.431160:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828062.431172:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564225920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828062.431184:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564226048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828062.431200:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828062.431205:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828062.431208:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828062.431213:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3002 00000100:00080000:0.0:1787828062.431219:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828062.431223:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828062.431229:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828062.431244:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564226176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828062.431257:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828062.431268:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787828062.431269:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828062.431276:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828062.431285:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3002 00000100:00080000:0.0:1787828062.431293:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828062.431298:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828062.431306:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828062.431322:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564226304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828062.431337:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828062/real 1787828062] req@ffff976882aeed00 x1874670564225920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828062.431339:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828062.431355:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828062.431360:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828062.431375:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3002 00000100:00080000:0.0:1787828062.431384:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828062.431388:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828062.431397:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564225920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828062.431398:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828062.431423:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828062.431434:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828062.431436:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564226432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828062.431444:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564225920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828062.431495:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564226048:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828062.431526:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828062.431543:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828062/real 1787828062] req@ffff976882aec3c0 x1874670564226048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828062.431559:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564226048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828062.431569:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828062.431574:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828062.431578:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564226048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828062.431588:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564226176:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828062.431609:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828062.431620:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828062/real 1787828062] req@ffff976882aee580 x1874670564226176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828062.431635:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564226176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828062.431644:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828062.431648:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828062.431651:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564226176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828062.431658:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564226304:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828062.431677:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828062.431688:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828062/real 1787828062] req@ffff976882aed680 x1874670564226304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828062.431702:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564226304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828062.431712:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828062.431715:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828062.431719:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564226304:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828062.431725:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564226432:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828062.431742:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828062.431752:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828062/real 1787828062] req@ffff976882aee940 x1874670564226432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828062.431767:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564226432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828062.431776:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828062.431780:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828062.431783:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564226432:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828062.686937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828062.686947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828062.687059:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828063.454985:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828063.710945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828063.710955:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828063.710993:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828064.478945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828064.734937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828064.734959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828064.735054:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828065.502992:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828065.759031:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828065.759043:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828065.759106:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828066.526972:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828066.783046:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828066.783049:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828066.783099:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828067.550976:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828067.550992:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828067.551005:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828067.551012:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828067.551027:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3007 00000100:00080000:0.0:1787828067.551041:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828067.551051:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828067.551098:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564226560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828067.551139:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828067.551144:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828067.551146:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828067.551151:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3007 00000100:00080000:0.0:1787828067.551156:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828067.551161:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828067.551174:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828067.551192:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564226688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828067.551209:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564226560:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828067.551214:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828067.551234:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828067.551237:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828067.551243:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3007 00000100:00080000:0.0:1787828067.551247:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828067.551255:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828067.551262:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828067.551278:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564226816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828067.551290:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828067.551294:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828067.551298:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828067.551304:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3007 00000100:00080000:0.0:1787828067.551310:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828067.551315:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828067.551321:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828067.551337:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564226944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828067.551351:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828067.551364:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828067.551367:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828067.551373:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828067.551381:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3007 00000100:00080000:0.0:1787828067.551386:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828067.551391:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828067.551402:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828067.551418:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564227072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828067.551464:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828067/real 1787828067] req@ffff976882aef0c0 x1874670564226560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828067.551499:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564226560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828067.551517:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828067.551523:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828067.551529:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564226560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828067.551568:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564226688:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828067.551600:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828067.551617:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828067/real 1787828067] req@ffff976882aee940 x1874670564226688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828067.551633:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564226688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828067.551644:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828067.551648:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828067.551653:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564226688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828067.551660:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564226816:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828067.551680:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828067.551691:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828067/real 1787828067] req@ffff976882aed680 x1874670564226816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828067.551706:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564226816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828067.551716:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828067.551719:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828067.551723:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564226816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828067.551730:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564226944:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828067.551752:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828067.551764:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828067/real 1787828067] req@ffff976882aee580 x1874670564226944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828067.551805:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564226944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828067.551816:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828067.551820:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828067.551857:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564226944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828067.551865:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564227072:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828067.551891:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828067.551906:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828067/real 1787828067] req@ffff976882aec3c0 x1874670564227072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828067.551920:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564227072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828067.551931:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828067.551935:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828067.551939:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564227072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828067.806996:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828067.807006:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828067.807034:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828068.574960:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828068.830983:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828068.831003:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828068.831051:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828069.598974:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828069.855015:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828069.855016:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828069.855064:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828070.622961:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828070.878987:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828070.879005:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828070.879162:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828071.646970:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828071.902902:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828071.902902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828071.902939:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828072.670897:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828072.670920:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828072.670929:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828072.670935:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828072.670946:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3012 00000100:00080000:0.0:1787828072.670967:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828072.670976:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828072.671029:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564227200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828072.671060:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828072.671063:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828072.671065:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828072.671069:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3012 00000100:00080000:0.0:1787828072.671072:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828072.671076:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828072.671088:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828072.671095:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564227200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828072.671104:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564227328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828072.671114:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828072.671117:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828072.671119:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828072.671122:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3012 00000100:00080000:0.0:1787828072.671125:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828072.671128:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828072.671131:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828072.671141:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564227456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828072.671148:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828072.671151:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828072.671153:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828072.671157:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3012 00000100:00080000:0.0:1787828072.671161:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828072.671165:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828072.671170:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828072.671179:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564227584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828072.671187:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828072.671195:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828072.671199:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828072.671201:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828072.671204:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3012 00000100:00080000:0.0:1787828072.671208:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828072.671211:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828072.671215:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828072.671224:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564227712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828072.671254:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828072/real 1787828072] req@ffff976882aeed00 x1874670564227200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828072.671272:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564227200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828072.671283:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828072.671288:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828072.671292:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564227200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828072.671320:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564227328:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828072.671340:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828072.671350:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828072/real 1787828072] req@ffff976882aec3c0 x1874670564227328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828072.671360:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564227328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828072.671367:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828072.671369:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828072.671372:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564227328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828072.671376:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564227456:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828072.671388:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828072.671395:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828072/real 1787828072] req@ffff976882aee580 x1874670564227456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828072.671404:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564227456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828072.671410:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828072.671412:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828072.671414:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564227456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828072.671436:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564227584:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828072.671454:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828072.671464:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828072/real 1787828072] req@ffff976882aed680 x1874670564227584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828072.671474:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564227584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828072.671482:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828072.671485:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828072.671488:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564227584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828072.671494:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564227712:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828072.671506:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828072.671513:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828072/real 1787828072] req@ffff976882aee940 x1874670564227712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828072.671521:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564227712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828072.671542:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828072.671544:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828072.671547:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564227712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828072.926920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828072.926920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828072.926983:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828073.694965:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828073.950955:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828073.950955:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828073.951006:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828074.718970:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828074.974909:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828074.974909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828074.974976:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828075.742985:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828075.998932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828075.998960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828075.999016:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828076.767015:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828077.022958:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828077.023009:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828077.023037:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828077.791036:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828077.791052:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828077.791060:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828077.791077:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3017 00000100:00080000:2.0:1787828077.791094:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828077.791107:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828077.791139:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828077.791176:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564227840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828077.791222:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828077.791227:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828077.791230:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828077.791236:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3017 00000100:00080000:2.0:1787828077.791240:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828077.791248:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828077.791265:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828077.791285:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564227968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828077.791298:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564227840:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828077.791334:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828077.791341:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828077.791344:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828077.791353:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3017 00000100:00080000:2.0:1787828077.791358:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828077.791364:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828077.791373:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828077.791391:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564228096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828077.791405:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828077.791408:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828077.791425:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828077.791429:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828077.791440:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3017 00000100:00080000:2.0:1787828077.791454:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828077.791461:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828077.791468:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828077.791485:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828077/real 1787828077] req@ffff976889cf8f00 x1874670564227840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828077.791491:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564228224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828077.791567:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828077.791571:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828077.791583:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828077.791609:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3017 00000100:00080000:2.0:1787828077.791617:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828077.791625:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564227840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828077.791632:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828077.791653:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828077.791673:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828077.791684:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828077.791696:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564228352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828077.791725:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564227840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828077.791789:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564227968:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828077.791885:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828077.791905:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828077/real 1787828077] req@ffff976889cfb0c0 x1874670564227968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828077.791932:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564227968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828077.791943:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828077.791948:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828077.791953:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564227968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828077.791961:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564228096:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828077.791998:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828077.792021:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828077/real 1787828077] req@ffff976889cf8000 x1874670564228096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828077.792045:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564228096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828077.792062:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828077.792070:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828077.792077:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564228096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828077.792091:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564228224:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828077.792129:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828077.792146:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828077/real 1787828077] req@ffff976889cfbc00 x1874670564228224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828077.792161:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564228224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828077.792181:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828077.792185:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828077.792189:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564228224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828077.792198:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564228352:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828077.792219:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828077.792230:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828077/real 1787828077] req@ffff976889cfa940 x1874670564228352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828077.792244:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564228352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828077.792254:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828077.792258:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828077.792261:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564228352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828078.046940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828078.046996:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828078.047071:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828078.814968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828079.070979:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828079.070996:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828079.071048:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828079.838940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828080.094952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828080.094955:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828080.095013:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828080.862996:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828081.119011:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828081.119011:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828081.119056:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828081.886934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828082.142978:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828082.142989:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828082.142991:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828082.910970:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828082.911076:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828082.911096:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828082.911104:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828082.911124:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3022 00000100:00080000:2.0:1787828082.911146:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828082.911162:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828082.911233:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564228480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828082.911282:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828082.911287:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828082.911290:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828082.911296:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3022 00000100:00080000:2.0:1787828082.911304:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828082.911311:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828082.911338:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828082.911360:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564228608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828082.911371:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564228480:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828082.911404:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828082.911409:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828082.911411:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828082.911429:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3022 00000100:00080000:2.0:1787828082.911433:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828082.911438:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828082.911444:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828082.911459:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564228736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828082.911470:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828082.911474:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828082.911477:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828082.911484:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3022 00000100:00080000:2.0:1787828082.911493:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828082.911499:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828082.911506:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828082.911523:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564228864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828082.911534:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828082.911568:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828082.911586:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828082.911591:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828082.911603:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3022 00000100:00080000:2.0:1787828082.911611:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828082.911629:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828082.911636:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828082.911645:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828082/real 1787828082] req@ffff976889cfa580 x1874670564228480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828082.911654:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564228992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828082.911766:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564228480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828082.911803:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828082.911811:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828082.911817:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564228480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828082.911948:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564228608:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828082.911988:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828082.912005:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828082/real 1787828082] req@ffff976889cfa1c0 x1874670564228608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828082.912020:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564228608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828082.912031:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828082.912035:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828082.912038:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564228608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828082.912064:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564228736:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828082.912088:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828082.912100:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828082/real 1787828082] req@ffff976889cf92c0 x1874670564228736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828082.912113:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564228736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828082.912123:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828082.912127:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828082.912130:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564228736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828082.912137:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564228864:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828082.912156:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828082.912167:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828082/real 1787828082] req@ffff976889cf9e00 x1874670564228864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828082.912186:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564228864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828082.912218:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828082.912224:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828082.912231:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564228864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828082.912244:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564228992:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828082.912286:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828082.912309:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828082/real 1787828082] req@ffff976889cfb480 x1874670564228992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828082.912342:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564228992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828082.912351:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828082.912355:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828082.912359:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564228992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828083.166919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828083.166942:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828083.167025:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828083.934930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828084.190997:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828084.191011:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828084.191047:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828084.958893:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828085.214967:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828085.214984:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828085.214999:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828085.982930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828086.238981:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828086.238981:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828086.239000:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828087.006946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828087.262945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828087.262957:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828087.263013:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828088.030897:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828088.030941:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828088.030948:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828088.030953:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828088.030962:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3027 00000100:00080000:0.0:1787828088.030973:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828088.030980:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828088.031019:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564229120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828088.031058:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828088.031061:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828088.031063:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828088.031067:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3027 00000100:00080000:0.0:1787828088.031071:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828088.031074:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828088.031084:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828088.031097:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564229248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828088.031106:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828088.031109:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828088.031111:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828088.031114:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3027 00000100:00080000:0.0:1787828088.031116:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828088.031121:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828088.031126:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828088.031134:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564229376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828088.031142:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828088.031145:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828088.031147:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828088.031150:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3027 00000100:00080000:0.0:1787828088.031153:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828088.031156:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828088.031161:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828088.031165:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564229120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828088.031173:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564229504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828088.031184:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828088.031186:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828088.031188:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828088.031191:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3027 00000100:00080000:0.0:1787828088.031195:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828088.031198:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828088.031203:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828088.031214:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564229632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828088.031244:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828088.031284:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828088/real 1787828088] req@ffff976882aef0c0 x1874670564229120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828088.031301:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564229120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828088.031318:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828088.031322:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828088.031326:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564229120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828088.031333:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564229248:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828088.031351:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828088.031362:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828088/real 1787828088] req@ffff976882aee940 x1874670564229248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828088.031394:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564229248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828088.031405:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828088.031408:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828088.031412:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564229248:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828088.031420:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564229376:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828088.031442:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828088.031454:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828088/real 1787828088] req@ffff976882aed680 x1874670564229376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828088.031467:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564229376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828088.031478:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828088.031481:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828088.031485:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564229376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828088.031519:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564229504:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828088.031540:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828088.031553:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828088/real 1787828088] req@ffff976882aee580 x1874670564229504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828088.031576:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564229504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828088.031623:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828088.031645:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828088.031652:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564229504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828088.031661:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564229632:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828088.031691:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828088.031707:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828088/real 1787828088] req@ffff976882aec3c0 x1874670564229632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828088.031719:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564229632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828088.031726:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828088.031729:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828088.031732:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564229632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828088.286965:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828088.286975:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828088.287023:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828089.054948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828089.310936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828089.310977:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828089.311004:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828090.078942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787828090.334937:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828090.334986:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828090.335010:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828091.102985:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828091.358942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828091.359011:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828091.359138:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828092.126886:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828092.382896:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828092.382897:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828092.382922:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828093.150915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828093.150917:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828093.150927:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828093.150932:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828093.150942:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3033 00000100:00080000:0.0:1787828093.150951:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828093.150963:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828093.150995:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564229760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828093.151022:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828093.151025:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828093.151027:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828093.151030:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3033 00000100:00080000:0.0:1787828093.151034:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828093.151037:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828093.151047:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828093.151053:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564229760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828093.151064:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564229888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828093.151076:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828093.151079:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828093.151080:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828093.151085:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3033 00000100:00080000:0.0:1787828093.151088:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828093.151091:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828093.151096:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828093.151116:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564230016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828093.151128:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828093.151129:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828093.151138:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828093.151141:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828093.151148:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3033 00000100:00080000:0.0:1787828093.151221:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828093.151227:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828093.151232:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828093.151247:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828093/real 1787828093] req@ffff976882aeed00 x1874670564229760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828093.151255:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564230144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828093.151267:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828093.151270:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828093.151272:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828093.151274:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3033 00000100:00080000:0.0:1787828093.151277:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828093.151279:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564229760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828093.151283:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828093.151291:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828093.151299:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564230272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828093.151394:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828093.151399:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828093.151403:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564229760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828093.151432:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564229888:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828093.151496:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828093.151510:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828093/real 1787828093] req@ffff976882aec3c0 x1874670564229888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828093.151538:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564229888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828093.151549:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828093.151554:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828093.151559:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564229888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828093.151567:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564230016:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828093.151632:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828093.151649:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828093/real 1787828093] req@ffff976882aee580 x1874670564230016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828093.151679:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564230016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828093.151685:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828093.151688:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828093.151691:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564230016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828093.151695:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564230144:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828093.151713:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828093.151721:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828093/real 1787828093] req@ffff976882aed680 x1874670564230144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828093.151746:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564230144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828093.151753:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828093.151756:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828093.151759:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564230144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828093.151763:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564230272:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828093.151778:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828093.151786:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828093/real 1787828093] req@ffff976882aee940 x1874670564230272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828093.151814:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564230272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828093.151822:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828093.151825:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828093.151827:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564230272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828093.406941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828093.406944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828093.407063:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828094.174899:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828094.430929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828094.430990:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828094.431020:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828095.198948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828095.454907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828095.454929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828095.455032:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828096.222931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828096.478933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828096.478948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828096.478988:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828097.246897:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828097.502914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828097.502958:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828097.503019:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828098.271013:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828098.271026:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828098.271032:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828098.271047:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3038 00000100:00100000:3.0:1787828098.271055:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828098.271061:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828098.271085:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828098.271139:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564230400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828098.271206:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828098.271213:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828098.271216:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828098.271224:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3038 00000100:00080000:0.0:1787828098.271231:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828098.271241:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828098.271258:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828098.271306:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564230528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828098.271322:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564230400:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828098.271366:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828098.271371:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828098.271374:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828098.271380:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3038 00000100:00080000:0.0:1787828098.271386:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828098.271393:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828098.271400:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828098.271420:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564230656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828098.271441:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828098.271445:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828098.271448:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828098.271453:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3038 00000100:00080000:0.0:1787828098.271470:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828098.271475:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828098.271481:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828098.271508:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564230784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828098.271515:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828098.271542:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828098.271547:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828098.271550:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828098.271555:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3038 00000100:00080000:0.0:1787828098.271593:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828098.271599:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828098.271608:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828098.271625:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564230912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828098.271696:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828098/real 1787828098] req@ffff976882aef0c0 x1874670564230400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828098.271756:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564230400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828098.271779:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828098.271788:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828098.271796:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564230400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828098.271916:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564230528:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828098.271973:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828098.272016:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828098/real 1787828098] req@ffff976882aee940 x1874670564230528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828098.272041:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564230528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828098.272058:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828098.272066:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828098.272072:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564230528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828098.272086:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564230656:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828098.272127:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828098.272150:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828098/real 1787828098] req@ffff976882aed680 x1874670564230656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828098.272193:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564230656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828098.272213:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828098.272221:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828098.272228:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564230656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828098.272242:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564230784:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828098.272283:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828098.272321:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828098/real 1787828098] req@ffff976882aee580 x1874670564230784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828098.272346:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564230784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828098.272364:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828098.272371:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828098.272378:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564230784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828098.272389:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564230912:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828098.272421:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828098.272439:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828098/real 1787828098] req@ffff976882aec3c0 x1874670564230912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828098.272463:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564230912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828098.272479:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828098.272503:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828098.272510:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564230912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828098.526971:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828098.526976:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828098.527050:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828099.294894:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828099.551001:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828099.551002:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828099.551054:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828100.318958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787828100.574951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828100.574956:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828100.575050:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828101.342890:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828101.598900:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828101.598949:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828101.598951:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828102.366956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828102.622944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828102.622969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828102.623012:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828103.390948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828103.390978:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828103.390988:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828103.390994:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828103.391009:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3043 00000100:00080000:0.0:1787828103.391023:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828103.391047:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828103.391099:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564231040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828103.391131:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828103.391136:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828103.391139:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828103.391144:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3043 00000100:00080000:0.0:1787828103.391148:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828103.391152:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828103.391165:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828103.391182:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564231168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828103.391198:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564231040:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828103.391202:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828103.391213:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828103.391216:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828103.391221:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3043 00000100:00080000:0.0:1787828103.391225:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828103.391229:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828103.391235:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828103.391249:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564231296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828103.391270:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828103.391274:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828103.391278:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828103.391283:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3043 00000100:00080000:0.0:1787828103.391287:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828103.391296:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:3.0:1787828103.391300:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:0.0:1787828103.391305:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828103.391324:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564231424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828103.391336:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828103.391340:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828103.391342:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828103.391348:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3043 00000100:00080000:0.0:1787828103.391353:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828103.391357:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828103.391362:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828103.391375:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564231552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828103.391385:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828103/real 1787828103] req@ffff976882aeed00 x1874670564231040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828103.391428:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564231040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828103.391452:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828103.391462:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828103.391472:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564231040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828103.391509:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564231168:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828103.391578:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828103.391604:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828103/real 1787828103] req@ffff976882aec3c0 x1874670564231168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828103.391630:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564231168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828103.391648:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828103.391655:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828103.391661:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564231168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828103.391674:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564231296:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828103.391721:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828103.391738:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828103/real 1787828103] req@ffff976882aee580 x1874670564231296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828103.391754:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564231296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828103.391765:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828103.391769:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828103.391773:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564231296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828103.391781:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564231424:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828103.391803:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828103.391814:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828103/real 1787828103] req@ffff976882aed680 x1874670564231424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828103.391863:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564231424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828103.391874:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828103.391878:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828103.391882:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564231424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828103.391889:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564231552:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828103.391912:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828103.391924:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828103/real 1787828103] req@ffff976882aee940 x1874670564231552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828103.391948:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564231552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828103.391957:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828103.391961:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828103.391964:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564231552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828103.646936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828103.646936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828103.647033:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828104.414930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828104.670993:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828104.671033:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828104.671047:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828105.438981:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828105.694962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828105.694989:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828105.695104:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828106.462958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828106.719010:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828106.719011:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828106.719101:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828107.487012:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828107.742950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828107.742985:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828107.743127:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828108.511005:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828108.511040:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828108.511063:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828108.511069:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828108.511084:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3048 00000100:00080000:2.0:1787828108.511098:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828108.511107:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828108.511166:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564231680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828108.511208:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828108.511213:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828108.511215:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828108.511220:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3048 00000100:00080000:2.0:1787828108.511225:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828108.511232:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828108.511257:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828108.511275:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564231808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828108.511288:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564231680:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828108.511307:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828108.511323:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828108.511329:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828108.511336:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3048 00000100:00080000:2.0:1787828108.511340:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828108.511344:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828108.511352:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828108.511376:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564231936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828108.511390:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828108.511394:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828108.511397:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828108.511402:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3048 00000100:00080000:2.0:1787828108.511427:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828108.511434:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828108.511442:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828108.511456:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564232064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828108.511468:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828108.511472:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828108.511475:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828108.511483:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3048 00000100:00080000:2.0:1787828108.511526:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828108.511536:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828108.511544:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828108.511560:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564231808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828108.511570:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564232192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828108.511612:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564231936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828108.511634:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564232064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828108.511655:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564232192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828108.766992:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828108.766992:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828108.767155:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828109.534948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828109.790944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828109.790979:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828109.790982:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828110.558990:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828110.814946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828110.814999:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828110.815096:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828111.582925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:1.0:1787828111.583183:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:1.0:1787828111.583251:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787828111.583273:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787828111.583327:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787828111.583344:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787828111.583361:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787828111.583382:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:3.0:1787828111.583424:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828108/real 1787828111] req@ffff976889cfa1c0 x1874670564232192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828111.583497:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564232192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828111.583524:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828111.583532:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828111.583540:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564232192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828111.583553:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828108/real 1787828111] req@ffff976889cfb480 x1874670564231808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828111.583581:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564231808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828111.583597:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828111.583604:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828111.583610:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564231808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828111.583623:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828108/real 1787828111] req@ffff976889cf9e00 x1874670564231936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828111.583647:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564231936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828111.583664:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828111.583670:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828111.583677:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564231936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828111.583690:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828108/real 1787828111] req@ffff976889cf92c0 x1874670564232064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828111.583716:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564232064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828111.583730:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828111.583735:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828111.583740:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564232064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828111.583754:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828108/real 1787828111] req@ffff976889cfad00 x1874670564231680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828111.583778:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564231680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828111.583795:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828111.583801:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828111.583807:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564231680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828111.838972:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828111.838972:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828111.839028:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828112.606941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828112.862960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828112.862965:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828112.863134:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828113.630978:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828113.630998:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828113.631010:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828113.631015:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828113.631045:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3053 00000100:00080000:2.0:1787828113.631058:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828113.631067:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828113.631115:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564232320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828113.631151:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828113.631173:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828113.631175:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828113.631180:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3053 00000100:00080000:2.0:1787828113.631186:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828113.631192:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828113.631205:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828113.631210:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564232320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828113.631226:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564232448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828113.631246:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828113.631252:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828113.631257:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828113.631264:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3053 00000100:00080000:2.0:1787828113.631289:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828113.631298:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828113.631307:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828113.631318:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828113.631336:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564232576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828113.631373:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828113.631378:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828113.631381:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828113.631389:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3053 00000100:00080000:2.0:1787828113.631396:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828113.631405:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828113.631414:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828113.631435:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564232704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828113.631496:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828113.631504:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828113.631507:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828113.631515:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3053 00000100:00080000:2.0:1787828113.631521:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828113.631528:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828113.631535:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828113/real 1787828113] req@ffff976889cfa580 x1874670564232320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828113.631541:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828113.631583:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564232832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828113.631590:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564232320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828113.631613:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828113.631623:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828113.631633:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564232320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828113.631726:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564232448:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828113.631770:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828113.631788:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828113/real 1787828113] req@ffff976889cfa940 x1874670564232448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828113.631804:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564232448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828113.631815:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828113.631820:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828113.631859:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564232448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828113.631974:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564232576:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828113.632008:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828113.632024:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828113/real 1787828113] req@ffff976889cfbc00 x1874670564232576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828113.632039:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564232576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828113.632049:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828113.632054:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828113.632058:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564232576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828113.632066:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564232704:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828113.632093:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828113.632107:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828113/real 1787828113] req@ffff976889cf8000 x1874670564232704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828113.632132:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564232704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828113.632145:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828113.632150:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828113.632155:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564232704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828113.632165:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564232832:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828113.632191:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828113.632213:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828113/real 1787828113] req@ffff976889cfb0c0 x1874670564232832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828113.632228:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564232832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828113.632238:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828113.632243:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828113.632247:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564232832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828113.886952:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828113.887007:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828113.887110:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828114.654965:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828114.910975:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828114.910981:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828114.911022:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828115.678954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828115.934916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828115.934942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828115.935022:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828116.702926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828116.958990:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828116.959002:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828116.959093:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828117.726968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828117.982972:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828117.982973:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828117.983125:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828118.750962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828118.751017:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828118.751028:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828118.751035:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828118.751051:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3058 00000100:00080000:1.0:1787828118.751070:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828118.751082:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828118.751149:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564232960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828118.751193:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828118.751199:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828118.751201:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828118.751219:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3058 00000100:00080000:1.0:1787828118.751225:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828118.751231:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828118.751248:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828118.751256:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564232960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828118.751271:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564233088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828118.751288:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828118.751292:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828118.751295:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828118.751300:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3058 00000100:00080000:1.0:1787828118.751304:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828118.751308:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828118.751314:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828118.751331:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564233216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828118.751342:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828118.751347:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828118.751350:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828118.751355:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3058 00000100:00080000:1.0:1787828118.751361:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828118.751365:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828118.751371:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787828118.751380:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828118.751389:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564233344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828118.751404:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828118.751409:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828118.751412:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828118.751443:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3058 00000100:00080000:1.0:1787828118.751449:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828118.751455:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828118.751463:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828118.751484:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564233472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828118.751497:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828118/real 1787828118] req@ffff976889efc780 x1874670564232960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828118.751534:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564232960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828118.751552:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828118.751558:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828118.751564:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564232960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828118.751601:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564233088:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828118.751651:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828118.751669:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828118/real 1787828118] req@ffff976889eff480 x1874670564233088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828118.751685:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564233088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828118.751697:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828118.751701:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828118.751705:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564233088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828118.751714:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564233216:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828118.751734:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828118.751745:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828118/real 1787828118] req@ffff976889efe580 x1874670564233216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828118.751769:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564233216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828118.751780:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828118.751785:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828118.751788:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564233216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828118.751795:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564233344:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828118.751816:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828118.751879:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828118/real 1787828118] req@ffff976889efe1c0 x1874670564233344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828118.751906:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564233344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828118.751924:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828118.751930:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828118.751936:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564233344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828118.751945:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564233472:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828118.751977:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828118.751992:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828118/real 1787828118] req@ffff976889efcb40 x1874670564233472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828118.752007:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564233472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828118.752026:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828118.752030:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828118.752034:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564233472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828119.006962:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828119.006972:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828119.006992:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828119.774981:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828120.031044:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828120.031088:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828120.031162:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828120.798955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828121.054894:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828121.054925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828121.054954:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828121.822943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828122.078992:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828122.079043:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828122.079173:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828122.847028:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828123.102989:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828123.102995:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828123.103047:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828123.871065:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828123.871067:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828123.871081:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828123.871096:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828123.871116:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3063 00000100:00080000:2.0:1787828123.871145:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828123.871151:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828123.871187:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564233600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828123.871240:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828123.871244:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828123.871247:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828123.871281:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3063 00000100:00080000:2.0:1787828123.871284:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828123.871287:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828123.871296:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828123.871312:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564233728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828123.871312:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564233600:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828123.871326:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828123.871329:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828123.871330:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828123.871333:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3063 00000100:00080000:2.0:1787828123.871336:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828123.871338:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828123.871341:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828123.871350:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564233856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828123.871357:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828123.871359:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828123.871361:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828123.871363:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3063 00000100:00080000:2.0:1787828123.871367:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828123.871370:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828123.871373:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787828123.871377:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828123.871393:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564233984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828123.871401:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828123.871404:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828123.871406:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828123.871409:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3063 00000100:00080000:2.0:1787828123.871413:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828123.871416:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828123.871419:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828123.871428:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564234112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828123.871446:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828123/real 1787828123] req@ffff976889cf8f00 x1874670564233600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828123.871466:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564233600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828123.871476:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828123.871480:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828123.871483:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564233600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828123.871503:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564233728:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828123.871521:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828123.871530:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828123/real 1787828123] req@ffff976889cfb0c0 x1874670564233728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828123.871539:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564233728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828123.871545:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828123.871547:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828123.871549:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564233728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828123.871553:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564233856:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828123.871564:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828123.871570:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828123/real 1787828123] req@ffff976889cf8000 x1874670564233856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828123.871578:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564233856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828123.871583:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828123.871585:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828123.871587:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564233856:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828123.871590:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564233984:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828123.871601:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828123.871624:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828123/real 1787828123] req@ffff976889cfbc00 x1874670564233984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828123.871633:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564233984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828123.871644:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828123.871646:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828123.871648:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564233984:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828123.871652:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564234112:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828123.871665:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828123.871671:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828123/real 1787828123] req@ffff976889cfa940 x1874670564234112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828123.871679:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564234112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828123.871684:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828123.871686:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828123.871688:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564234112:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828124.126933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828124.126937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828124.127001:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828124.894902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828125.150974:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828125.151022:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828125.151031:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828125.918950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828126.174973:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828126.174988:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828126.175025:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828126.943014:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828127.198968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828127.199000:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828127.199008:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828127.966939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787828128.222980:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828128.222993:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828128.222994:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00080000:2.0:1787828128.990967:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828128.990989:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828128.990996:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828128.991010:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3068 00000100:00080000:2.0:1787828128.991025:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828128.991036:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828128.991046:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828128.991088:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564234240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828128.991132:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828128.991146:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828128.991149:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828128.991155:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3068 00000100:00080000:2.0:1787828128.991159:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828128.991164:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828128.991189:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828128.991208:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564234368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828128.991225:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828128.991230:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564234240:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828128.991232:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828128.991250:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828128.991259:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3068 00000100:00080000:2.0:1787828128.991263:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828128.991267:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828128.991273:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828128.991289:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564234496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828128.991302:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828128.991307:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828128.991309:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828128.991316:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3068 00000100:00080000:2.0:1787828128.991321:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828128.991325:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:1.0:1787828128.991333:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:2.0:1787828128.991335:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828128.991378:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564234624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828128.991392:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828128.991397:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828128.991400:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828128.991404:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3068 00000100:00080000:2.0:1787828128.991410:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828128.991415:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828128.991422:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828128.991436:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564234752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828128.991451:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828128/real 1787828128] req@ffff976889cfa580 x1874670564234240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828128.991480:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564234240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828128.991497:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828128.991503:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828128.991508:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564234240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828128.991567:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564234368:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828128.991611:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828128.991630:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828128/real 1787828128] req@ffff976889cfad00 x1874670564234368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828128.991650:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564234368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828128.991662:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828128.991667:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828128.991672:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564234368:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828128.991681:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564234496:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828128.991725:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828128.991744:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828128/real 1787828128] req@ffff976889cf92c0 x1874670564234496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828128.991765:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564234496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828128.991779:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828128.991783:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828128.991788:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564234496:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828128.991797:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564234624:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828128.991898:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828128.991928:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828128/real 1787828128] req@ffff976889cf9e00 x1874670564234624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828128.991944:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564234624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828128.991955:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828128.991959:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828128.991964:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564234624:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828128.991979:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564234752:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828128.992005:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828128.992017:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828128/real 1787828128] req@ffff976889cfb480 x1874670564234752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828128.992030:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564234752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828128.992041:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828128.992045:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828128.992048:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564234752:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828129.246945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828129.247047:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828129.247183:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828130.014945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828130.270965:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828130.271001:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828130.271030:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828131.038956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828131.294951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828131.295039:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828131.295059:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828132.062979:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828132.318941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828132.318960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828132.319045:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828133.086985:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828133.343006:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828133.343011:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828133.343040:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828134.110999:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828134.111016:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828134.111034:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828134.111043:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828134.111061:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3074 00000100:00080000:2.0:1787828134.111079:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828134.111092:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828134.111153:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564234880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828134.111207:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828134.111214:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828134.111218:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828134.111227:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3074 00000100:00080000:2.0:1787828134.111234:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828134.111242:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828134.111260:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828134.111287:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564235008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828134.111303:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828134.111341:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828134.111347:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828134.111356:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3074 00000100:00080000:2.0:1787828134.111365:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828134.111374:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564234880:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828134.111380:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828134.111400:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828134.111438:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564235136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828134.111454:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828134.111458:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828134.111462:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828134.111469:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3074 00000100:00080000:2.0:1787828134.111475:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828134.111487:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:1.0:1787828134.111489:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:2.0:1787828134.111500:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828134.111521:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564235264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828134.111534:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828134.111538:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828134.111541:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828134.111545:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3074 00000100:00080000:2.0:1787828134.111549:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828134.111552:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828134.111560:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828134.111578:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564235392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828134.111584:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828134/real 1787828134] req@ffff976889cfa1c0 x1874670564234880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828134.111624:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564234880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828134.111640:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828134.111646:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828134.111651:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564234880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828134.111662:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564235008:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828134.111692:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828134.111708:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828134/real 1787828134] req@ffff976889cfb480 x1874670564235008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828134.111723:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564235008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828134.111735:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828134.111739:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828134.111743:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564235008:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828134.111774:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564235136:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828134.111794:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828134.111806:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828134/real 1787828134] req@ffff976889cf9e00 x1874670564235136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828134.111820:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564235136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828134.111874:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828134.111878:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828134.111882:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564235136:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828134.111890:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564235264:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828134.111915:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828134.111929:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828134/real 1787828134] req@ffff976889cf92c0 x1874670564235264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828134.111944:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564235264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828134.111954:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828134.111957:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828134.111962:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564235264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828134.111969:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564235392:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828134.111987:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828134.111998:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828134/real 1787828134] req@ffff976889cfad00 x1874670564235392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828134.112011:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564235392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828134.112021:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828134.112025:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828134.112028:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564235392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828134.366921:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828134.366985:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828134.368024:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828135.134929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828135.390920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828135.390974:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828135.391032:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828136.158901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828136.414968:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828136.414975:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828136.415006:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828137.182991:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828137.438986:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828137.439039:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828137.439097:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828138.206957:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828138.462946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828138.462946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828138.463035:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828139.230987:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828139.230992:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828139.231006:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828139.231025:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828139.231039:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3079 00000100:00080000:0.0:1787828139.231053:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828139.231062:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828139.231108:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564235520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828139.231154:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828139.231162:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828139.231165:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828139.231174:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3079 00000100:00080000:0.0:1787828139.231182:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828139.231191:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828139.231220:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828139.231248:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564235648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828139.231301:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828139.231307:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828139.231311:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828139.231318:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3079 00000100:00080000:0.0:1787828139.231326:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828139.231333:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828139.231342:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828139.231349:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564235520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828139.231363:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564235776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828139.231381:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828139.231386:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828139.231389:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828139.231396:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3079 00000100:00080000:0.0:1787828139.231421:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828139.231429:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:1.0:1787828139.231436:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:0.0:1787828139.231437:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828139.231464:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564235904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828139.231481:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828139.231487:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828139.231490:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828139.231496:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3079 00000100:00080000:0.0:1787828139.231503:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828139.231509:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828139/real 1787828139] req@ffff976882aef0c0 x1874670564235520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828139.231510:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828139.231518:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828139.231536:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564236032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828139.231539:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564235520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828139.231560:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828139.231567:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828139.231576:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564235520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828139.231590:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564235648:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828139.231634:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828139.231666:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828139/real 1787828139] req@ffff976882aee940 x1874670564235648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828139.231685:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564235648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828139.231703:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828139.231710:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828139.231716:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564235648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828139.231770:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564235776:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828139.231820:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828139.231886:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828139/real 1787828139] req@ffff976882aed680 x1874670564235776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828139.231903:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564235776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828139.231914:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828139.231918:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828139.231922:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564235776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828139.231930:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564235904:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828139.231956:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828139.231969:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828139/real 1787828139] req@ffff976882aee580 x1874670564235904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828139.231983:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564235904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828139.231993:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828139.231997:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828139.232000:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564235904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828139.232007:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564236032:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828139.232025:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828139.232036:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828139/real 1787828139] req@ffff976882aec3c0 x1874670564236032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828139.232049:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564236032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828139.232059:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828139.232062:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828139.232066:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564236032:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828139.486962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828139.487010:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828139.487066:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828140.254928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787828140.510923:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828140.510933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828140.510935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828141.278932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828141.534915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828141.534915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828141.534975:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828142.302938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828142.558900:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828142.558901:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828142.558921:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828143.326950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828143.582951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828143.582952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828143.583069:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828144.350946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828144.350978:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828144.350989:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828144.350995:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828144.351008:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3084 00000100:00080000:1.0:1787828144.351023:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828144.351033:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828144.351082:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564236160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828144.351137:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828144.351144:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828144.351148:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828144.351159:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3084 00000100:00080000:1.0:1787828144.351169:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828144.351178:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828144.351187:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564236160:192.168.203.139@tcp:250:kworker.0 00000008:00100000:1.0:1787828144.351201:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828144.351230:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564236288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828144.351284:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828144.351289:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828144.351292:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828144.351298:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3084 00000100:00080000:1.0:1787828144.351305:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828144.351315:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:3.0:1787828144.351316:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787828144.351331:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828144.351352:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564236416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828144.351366:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828144.351372:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828144.351375:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828144.351383:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828144/real 1787828144] req@ffff976889effc00 x1874670564236160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828144.351387:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3084 00000100:00080000:1.0:1787828144.351398:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828144.351409:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828144.351432:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564236160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828144.351435:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1787828144.351458:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828144.351464:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564236544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828144.351471:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828144.351486:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564236160:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787828144.351492:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828144.351496:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828144.351498:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828144.351504:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3084 00000100:00080000:1.0:1787828144.351507:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828144.351511:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828144.351516:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828144.351528:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564236672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828144.351583:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564236288:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828144.351627:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828144.351644:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828144/real 1787828144] req@ffff976889efed00 x1874670564236288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828144.351661:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564236288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828144.351672:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828144.351676:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828144.351680:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564236288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828144.351698:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564236416:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828144.351720:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828144.351735:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828144/real 1787828144] req@ffff976889efc3c0 x1874670564236416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828144.351759:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564236416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828144.351778:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828144.351784:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828144.351790:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564236416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828144.351802:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564236544:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828144.351879:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828144.351898:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828144/real 1787828144] req@ffff976889efe940 x1874670564236544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828144.351915:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564236544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828144.351926:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828144.351930:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828144.351934:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564236544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828144.351941:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564236672:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828144.351964:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828144.351975:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828144/real 1787828144] req@ffff976889eff0c0 x1874670564236672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828144.351988:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564236672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828144.351998:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828144.352001:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828144.352005:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564236672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828144.606952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828144.606969:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828144.607029:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828145.374904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828145.630948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828145.630948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828145.631002:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828146.398905:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828146.654951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828146.654970:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828146.655102:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828147.422927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828147.678944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828147.678962:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828147.679086:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828148.446950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828148.702943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828148.702960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828148.703063:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828149.470912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828149.471035:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828149.471044:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828149.471050:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828149.471063:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3089 00000100:00080000:0.0:1787828149.471076:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828149.471086:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828149.471147:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564236800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828149.471198:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828149.471227:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828149.471239:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828149.471268:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3089 00000100:00080000:0.0:1787828149.471276:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828149.471285:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564236800:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828149.471287:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828149.471308:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828149.471358:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564236928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828149.471383:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828149.471391:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828149.471395:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828149.471403:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3089 00000800:00000100:3.0:1787828149.471431:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828149.471434:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828149.471445:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828149.471455:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828149.471473:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564237056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828149.471485:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828149.471489:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828149.471495:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828149.471495:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828149/real 1787828149] req@ffff976882aeed00 x1874670564236800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828149.471506:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3089 00000100:00080000:0.0:1787828149.471516:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828149.471525:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828149.471532:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828149.471547:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564236800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828149.471548:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564237184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828149.471598:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828149.471601:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828149.471609:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:0.0:1787828149.471616:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787828149.471617:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564236800:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828149.471624:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828149.471633:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3089 00000100:00080000:0.0:1787828149.471636:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828149.471640:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828149.471645:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828149.471659:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564236928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828149.471660:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564237312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828149.471726:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828149.471749:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828149/real 1787828149] req@ffff976882aec3c0 x1874670564236928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828149.471764:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564236928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828149.471774:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828149.471778:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828149.471782:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564236928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828149.471789:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564237056:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828149.471811:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828149.471822:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828149/real 1787828149] req@ffff976882aee580 x1874670564237056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828149.471841:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564237056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828149.471850:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828149.471906:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828149.471911:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564237056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828149.471919:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564237184:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828149.471951:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828149.471973:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828149/real 1787828149] req@ffff976882aed680 x1874670564237184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828149.472008:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564237184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828149.472025:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828149.472032:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828149.472040:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564237184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828149.472070:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564237312:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828149.472100:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828149.472121:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828149/real 1787828149] req@ffff976882aee940 x1874670564237312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828149.472136:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564237312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828149.472155:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828149.472159:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828149.472164:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564237312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828149.726951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828149.726957:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828149.727102:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828150.494989:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828150.750985:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828150.751017:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828150.751121:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828151.518948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828151.774959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828151.774966:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828151.775097:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828152.542994:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828152.798933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828152.798943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828152.799046:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828153.566958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828153.822974:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828153.822974:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828153.823097:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828154.590961:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828154.590999:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828154.591021:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828154.591026:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828154.591039:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3094 00000100:00080000:2.0:1787828154.591053:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828154.591061:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828154.591118:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564237440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828154.591155:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828154.591159:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828154.591162:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828154.591193:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3094 00000100:00080000:2.0:1787828154.591200:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828154.591207:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828154.591220:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828154.591242:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564237440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828154.591244:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564237568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828154.591262:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828154.591275:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828154.591280:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828154.591286:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3094 00000100:00080000:2.0:1787828154.591290:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828154.591294:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828154.591299:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828154.591313:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564237696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828154.591324:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828154.591329:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828154.591331:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828154.591353:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3094 00000100:00080000:2.0:1787828154.591386:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828154.591391:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828154.591398:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787828154.591409:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828154.591417:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564237824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828154.591434:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828154.591439:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828154.591442:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828154.591468:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3094 00000100:00080000:2.0:1787828154.591483:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828154.591489:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828154.591495:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828154.591509:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564237952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828154.591523:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828154/real 1787828154] req@ffff976889cfa580 x1874670564237440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828154.591570:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564237440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828154.591592:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828154.591602:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828154.591611:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564237440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828154.591660:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564237568:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828154.591705:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828154.591733:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828154/real 1787828154] req@ffff976889cfa940 x1874670564237568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828154.591771:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564237568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828154.591795:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828154.591816:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828154.591821:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564237568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828154.591831:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564237696:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828154.591890:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828154.591902:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828154/real 1787828154] req@ffff976889cfbc00 x1874670564237696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828154.591934:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564237696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828154.591943:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828154.591947:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828154.591951:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564237696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828154.591960:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564237824:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828154.591995:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828154.592016:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828154/real 1787828154] req@ffff976889cf8000 x1874670564237824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828154.592049:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564237824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828154.592074:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828154.592079:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828154.592082:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564237824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828154.592091:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564237952:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828154.592118:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828154.592131:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828154/real 1787828154] req@ffff976889cfb0c0 x1874670564237952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828154.592144:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564237952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828154.592154:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828154.592157:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828154.592161:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564237952:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828154.846970:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828154.846997:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828154.847101:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828155.614941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828155.870986:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828155.870997:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828155.871115:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828156.638873:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828156.894936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828156.894936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828156.894979:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828157.662889:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828157.918900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828157.918947:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828157.918990:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828158.686931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828158.942945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828158.942973:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828158.943112:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828159.710938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828159.711006:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828159.711017:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828159.711023:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828159.711036:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3099 00000100:00080000:2.0:1787828159.711071:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828159.711081:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828159.711171:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564238080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828159.711226:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828159.711230:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828159.711232:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828159.711237:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3099 00000100:00080000:2.0:1787828159.711242:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828159.711248:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828159.711262:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828159.711272:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564238080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828159.711283:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564238208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828159.711296:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828159.711301:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828159.711303:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828159.711327:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3099 00000100:00080000:2.0:1787828159.711333:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828159.711338:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828159.711348:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828159.711373:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828159.711394:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564238336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828159.711422:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828159.711432:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828159/real 1787828159] req@ffff976889cf8f00 x1874670564238080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828159.711433:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828159.711444:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828159.711472:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3099 00000100:00100000:3.0:1787828159.711482:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564238080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828159.711485:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828159.711507:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:3.0:1787828159.711507:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828159.711536:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000008:00100000:2.0:1787828159.711544:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828159.711547:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564238080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828159.711578:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564238464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828159.711595:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564238208:192.168.203.139@tcp:38:kworker.0 00000100:00080000:2.0:1787828159.711596:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828159.711610:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828159.711613:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828159.711620:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3099 00000100:00080000:2.0:1787828159.711626:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828159.711631:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828159.711638:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828159.711653:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564238592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828159.711663:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828159.711698:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828159/real 1787828159] req@ffff976889cfb0c0 x1874670564238208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828159.711720:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564238208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828159.711733:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828159.711748:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828159.711753:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564238208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828159.711764:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564238336:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828159.711793:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828159.711804:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828159/real 1787828159] req@ffff976889cf8000 x1874670564238336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828159.711819:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564238336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828159.711871:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828159.711877:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828159.711881:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564238336:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828159.711900:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564238464:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828159.711922:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828159.711933:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828159/real 1787828159] req@ffff976889cfbc00 x1874670564238464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828159.711943:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564238464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828159.711950:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828159.711953:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828159.711956:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564238464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828159.711972:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564238592:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828159.711985:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828159.711992:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828159/real 1787828159] req@ffff976889cfa940 x1874670564238592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828159.712001:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564238592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828159.712008:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828159.712010:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828159.712013:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564238592:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828159.966880:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828159.966912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828159.966953:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828160.734950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828160.990912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828160.990912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828160.990936:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828161.758937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828162.014880:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828162.014888:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828162.014944:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828162.782971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828163.038894:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828163.038894:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828163.038969:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828163.806936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828164.062935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828164.062987:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828164.063135:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828164.831004:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828164.831018:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828164.831024:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828164.831037:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3104 00000100:00080000:2.0:1787828164.831050:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828164.831059:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828164.831127:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564238720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828164.831173:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828164.831177:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828164.831185:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828164.831188:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828164.831196:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3104 00000100:00080000:2.0:1787828164.831204:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828164.831210:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828164.831223:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828164.831246:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564238720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828164.831248:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564238848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828164.831284:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828164.831291:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828164.831295:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828164.831304:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3104 00000100:00080000:2.0:1787828164.831313:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828164.831323:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828164.831334:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828164.831365:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564238976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828164.831380:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828164.831381:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828164.831384:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828164.831391:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828164.831397:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3104 00000100:00080000:2.0:1787828164.831403:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828164.831424:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828164.831434:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828164.831457:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828164/real 1787828164] req@ffff976889cfa580 x1874670564238720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828164.831461:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564239104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828164.831504:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828164.831507:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828164.831510:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828164.831514:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3104 00000100:00080000:2.0:1787828164.831519:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828164.831524:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828164.831530:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828164.831543:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564239232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828164.831620:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564238720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828164.831953:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828164.831977:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828164.831987:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564238720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828164.832028:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564238848:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828164.832090:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828164.832155:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828164/real 1787828164] req@ffff976889cfad00 x1874670564238848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828164.832183:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564238848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828164.832202:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828164.832209:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828164.832217:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564238848:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828164.832230:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564238976:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828164.832273:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828164.832298:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828164/real 1787828164] req@ffff976889cf92c0 x1874670564238976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828164.832322:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564238976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828164.832340:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828164.832347:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828164.832353:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564238976:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828164.832366:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564239104:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828164.832403:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828164.832426:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828164/real 1787828164] req@ffff976889cf9e00 x1874670564239104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828164.832451:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564239104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828164.832467:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828164.832473:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828164.832480:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564239104:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828164.832492:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564239232:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828164.832531:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828164.832554:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828164/real 1787828164] req@ffff976889cfb480 x1874670564239232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828164.832578:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564239232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828164.832594:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828164.832600:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828164.832606:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564239232:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828165.087006:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828165.087007:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828165.087117:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828165.854972:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828166.110955:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828166.110956:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828166.111059:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828166.878982:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828167.135009:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828167.135014:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828167.135146:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828167.902950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828168.158991:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828168.158995:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828168.159131:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828168.927027:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828169.182942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828169.182943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828169.183032:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828169.950958:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828169.950965:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828169.950972:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828169.950979:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828169.950991:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3109 00000100:00080000:2.0:1787828169.951005:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828169.951016:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828169.951083:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564239360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828169.951139:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828169.951146:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828169.951150:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828169.951159:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3109 00000100:00080000:2.0:1787828169.951166:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828169.951175:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828169.951204:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564239360:192.168.203.139@tcp:250:kworker.0 00000008:00100000:2.0:1787828169.951206:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828169.951234:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564239488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828169.951248:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828169.951251:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828169.951253:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828169.951257:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3109 00000100:00080000:2.0:1787828169.951264:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828169.951268:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828169.951273:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828169.951279:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828169.951287:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564239616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828169.951299:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828169.951302:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828169.951304:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828169.951307:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3109 00000100:00080000:2.0:1787828169.951310:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828169.951314:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828169.951319:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828169.951327:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828169/real 1787828169] req@ffff976889cfa1c0 x1874670564239360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828169.951334:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564239744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828169.951361:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828169.951364:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828169.951366:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828169.951369:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3109 00000100:00080000:2.0:1787828169.951371:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828169.951375:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828169.951380:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564239360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828169.951381:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828169.951398:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564239872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828169.951400:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828169.951414:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828169.951423:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564239360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828169.951450:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564239488:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828169.951472:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828169.951484:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828169/real 1787828169] req@ffff976889cfb480 x1874670564239488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828169.951502:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564239488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828169.951511:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828169.951514:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828169.951517:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564239488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828169.951522:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564239616:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828169.951536:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828169.951555:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828169/real 1787828169] req@ffff976889cf9e00 x1874670564239616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828169.951565:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564239616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828169.951573:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828169.951576:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828169.951578:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564239616:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828169.951583:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564239744:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828169.951604:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828169.951613:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828169/real 1787828169] req@ffff976889cf92c0 x1874670564239744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828169.951623:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564239744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828169.951636:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828169.951638:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828169.951641:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564239744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828169.951646:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564239872:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828169.951659:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828169.951666:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828169/real 1787828169] req@ffff976889cfad00 x1874670564239872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828169.951676:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564239872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828169.951683:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828169.951686:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828169.951688:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564239872:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828170.206937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828170.206962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828170.207035:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828170.974939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828171.231000:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828171.231000:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828171.231094:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828171.998939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828172.254956:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828172.254961:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828172.255118:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828173.022959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828173.278974:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828173.278974:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828173.279102:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828174.046916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828174.302941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828174.302943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828174.303101:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828175.070958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828175.071058:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828175.071067:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828175.071084:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828175.071095:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3114 00000100:00080000:1.0:1787828175.071108:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828175.071116:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828175.071163:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564240000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828175.071203:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828175.071208:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828175.071210:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828175.071215:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3114 00000100:00080000:1.0:1787828175.071219:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828175.071224:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828175.071236:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828175.071254:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564240128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828175.071266:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828175.071271:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828175.071273:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828175.071278:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3114 00000100:00080000:1.0:1787828175.071281:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828175.071286:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828175.071292:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828175.071306:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564240000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828175.071313:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564240256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828175.071343:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828175.071348:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828175.071352:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828175.071362:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3114 00000100:00080000:1.0:1787828175.071369:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828175.071378:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828175.071389:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828175.071412:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564240384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828175.071444:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828175.071450:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828175.071453:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828175.071460:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3114 00000100:00080000:1.0:1787828175.071467:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828175.071476:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828175.071485:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828175.071505:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564240128:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828175.071516:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564240512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828175.071569:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564240256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828175.071631:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564240384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828175.071663:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564240512:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828175.327006:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828175.327008:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828175.327128:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828176.094928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828176.351009:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828176.351050:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828176.351148:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828177.118975:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828177.374909:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828177.374909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828177.374959:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828178.142982:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:2.0:1787828178.143165:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:2.0:1787828178.143206:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787828178.143220:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787828178.143260:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787828178.143270:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787828178.143278:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787828178.143289:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:3.0:1787828178.143341:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828175/real 1787828178] req@ffff976889efc3c0 x1874670564240256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828178.143444:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564240256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828178.143464:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828178.143473:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828178.143480:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564240256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828178.143493:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828175/real 1787828178] req@ffff976889efed00 x1874670564240384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828178.143515:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564240384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828178.143528:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828178.143534:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828178.143539:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564240384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828178.143551:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828175/real 1787828178] req@ffff976889effc00 x1874670564240512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828178.143568:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564240512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828178.143580:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828178.143585:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828178.143590:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564240512:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828178.143601:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828175/real 1787828178] req@ffff976889eff0c0 x1874670564240000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828178.143622:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564240000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828178.143632:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828178.143637:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828178.143642:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564240000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828178.143651:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828175/real 1787828178] req@ffff976889efe940 x1874670564240128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828178.143667:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564240128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828178.143677:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828178.143683:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828178.143688:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564240128:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828178.398991:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828178.398991:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828178.399167:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828179.167190:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828179.422946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828179.422987:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828179.423055:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828180.190952:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828180.190967:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828180.190972:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828180.190986:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3120 00000100:00100000:0.0:1787828180.191002:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828180.191011:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828180.191020:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828180.191056:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564240640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828180.191083:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828180.191086:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828180.191089:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828180.191092:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3120 00000100:00080000:1.0:1787828180.191096:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828180.191099:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828180.191110:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828180.191123:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564240768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828180.191126:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564240640:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787828180.191138:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828180.191142:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828180.191144:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828180.191148:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3120 00000100:00080000:1.0:1787828180.191150:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828180.191153:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828180.191157:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828180.191172:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564240896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828180.191182:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828180.191187:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828180.191190:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828180.191195:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3120 00000100:00080000:1.0:1787828180.191202:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828180.191208:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828180.191215:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787828180.191215:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828180.191236:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564241024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828180.191249:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828180.191254:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828180.191257:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828180.191262:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3120 00000100:00080000:1.0:1787828180.191270:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828180.191270:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828180/real 1787828180] req@ffff976889efcb40 x1874670564240640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828180.191282:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828180.191294:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828180.191312:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564240640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828180.191314:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564241152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828180.191346:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828180.191359:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828180.191366:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564240640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828180.191412:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564240768:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828180.191466:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828180.191485:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828180/real 1787828180] req@ffff976889efe1c0 x1874670564240768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828180.191503:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564240768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828180.191514:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828180.191518:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828180.191523:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564240768:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828180.191532:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564240896:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828180.191560:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828180.191576:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828180/real 1787828180] req@ffff976889efe580 x1874670564240896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828180.191593:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564240896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828180.191604:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828180.191608:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828180.191612:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564240896:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828180.191621:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564241024:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828180.191648:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828180.191662:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828180/real 1787828180] req@ffff976889eff480 x1874670564241024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828180.191677:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564241024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828180.191687:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828180.191691:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828180.191694:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564241024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828180.191701:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564241152:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828180.191724:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828180.191737:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828180/real 1787828180] req@ffff976889efc780 x1874670564241152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828180.191752:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564241152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828180.191762:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828180.191766:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828180.191770:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564241152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828180.446970:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828180.446980:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828180.447054:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828181.215039:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828181.471019:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828181.471033:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828181.471073:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828182.239004:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828182.494956:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828182.494956:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828182.495118:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828183.262958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828183.519003:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828183.519003:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828183.519137:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828184.286958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828184.543026:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828184.543051:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828184.543053:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828185.310987:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828185.310997:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828185.311003:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828185.311015:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3125 00000100:00080000:2.0:1787828185.311029:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828185.311037:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828185.311098:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564241280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828185.311109:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828185.311149:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828185.311154:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828185.311158:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828185.311168:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3125 00000100:00080000:2.0:1787828185.311173:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828185.311179:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828185.311191:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828185.311193:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564241280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828185.311212:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564241408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828185.311225:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828185.311229:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828185.311233:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828185.311238:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3125 00000100:00080000:2.0:1787828185.311243:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828185.311248:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828185.311254:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828185.311267:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564241536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828185.311276:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828185.311277:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828185.311281:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828185.311283:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828185.311287:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3125 00000100:00080000:2.0:1787828185.311291:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828185.311294:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828185.311299:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828185.311310:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564241664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828185.311317:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828185.311320:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828185.311321:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828185.311324:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3125 00000100:00080000:2.0:1787828185.311326:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828185.311328:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828185/real 1787828185] req@ffff976889cfa580 x1874670564241280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828185.311331:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828185.311338:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828185.311349:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564241792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828185.311352:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564241280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828185.311379:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828185.311383:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828185.311389:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564241280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828185.311414:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564241408:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828185.311432:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828185.311442:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828185/real 1787828185] req@ffff976889cfa940 x1874670564241408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828185.311450:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564241408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828185.311457:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828185.311459:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828185.311462:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564241408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828185.311465:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564241536:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828185.311476:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828185.311482:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828185/real 1787828185] req@ffff976889cfbc00 x1874670564241536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828185.311490:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564241536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828185.311496:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828185.311497:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828185.311500:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564241536:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828185.311503:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564241664:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828185.311514:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828185.311521:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828185/real 1787828185] req@ffff976889cf8000 x1874670564241664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828185.311528:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564241664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828185.311534:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828185.311535:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828185.311537:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564241664:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828185.311540:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564241792:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828185.311550:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828185.311556:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828185/real 1787828185] req@ffff976889cfb0c0 x1874670564241792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828185.311563:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564241792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828185.311569:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828185.311570:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828185.311572:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564241792:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828185.566915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828185.566944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828185.566996:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828186.334895:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828186.590957:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828186.590987:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828186.590992:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828187.359073:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828187.614954:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828187.614992:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828187.615072:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828188.382930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828188.638979:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828188.638981:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828188.638985:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828189.406979:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828189.663019:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828189.663040:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828189.663100:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828190.430952:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828190.430981:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828190.430986:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828190.430999:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3130 00000100:00100000:0.0:1787828190.431001:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828190.431014:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828190.431044:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828190.431111:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564241920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828190.431155:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828190.431160:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828190.431163:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828190.431169:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3130 00000100:00080000:2.0:1787828190.431174:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828190.431179:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828190.431210:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828190.431232:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564242048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828190.431242:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828190.431249:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787828190.431251:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564241920:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828190.431253:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828190.431267:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3130 00000100:00080000:2.0:1787828190.431280:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828190.431285:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828190.431292:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828190.431306:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564242176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828190.431315:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828190.431318:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828190.431320:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828190.431324:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3130 00000100:00080000:2.0:1787828190.431328:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828190.431332:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828190.431337:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828190.431347:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564242304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828190.431362:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828190.431369:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828190.431373:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828190.431394:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828190.431402:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3130 00000100:00080000:2.0:1787828190.431410:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828190.431417:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828190.431428:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828190.431442:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828190/real 1787828190] req@ffff976889cf8f00 x1874670564241920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828190.431458:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564242432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828190.431476:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564241920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828190.431503:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828190.431511:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828190.431518:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564241920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828190.431574:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564242048:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828190.431599:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828190.431613:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828190/real 1787828190] req@ffff976889cfb0c0 x1874670564242048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828190.431626:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564242048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828190.431636:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828190.431639:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828190.431642:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564242048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828190.431648:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564242176:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828190.431672:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828190.431689:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828190/real 1787828190] req@ffff976889cf8000 x1874670564242176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828190.431709:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564242176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828190.431751:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828190.431755:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828190.431759:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564242176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828190.431766:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564242304:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828190.431792:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828190.431809:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828190/real 1787828190] req@ffff976889cfbc00 x1874670564242304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828190.431821:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564242304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828190.431870:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828190.431874:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828190.431877:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564242304:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828190.431884:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564242432:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828190.431904:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828190.431915:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828190/real 1787828190] req@ffff976889cfa940 x1874670564242432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828190.431955:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564242432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828190.431964:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828190.431976:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828190.431980:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564242432:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828190.687048:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828190.687065:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828190.687081:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828191.454959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828191.710974:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828191.710988:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828191.711003:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828192.478887:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828192.735005:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828192.735023:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828192.735032:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828193.502946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828193.758929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828193.758965:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828193.759004:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828194.527055:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828194.783006:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828194.783015:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828194.783054:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828195.550973:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828195.550984:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828195.550996:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828195.551002:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828195.551018:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3135 00000100:00080000:2.0:1787828195.551033:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828195.551041:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828195.551094:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564242560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828195.551131:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828195.551135:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828195.551136:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828195.551140:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3135 00000100:00080000:2.0:1787828195.551144:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828195.551147:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828195.551157:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828195.551170:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564242688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828195.551180:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564242560:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828195.551180:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828195.551189:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828195.551191:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828195.551195:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3135 00000100:00080000:2.0:1787828195.551198:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828195.551200:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828195.551204:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828195.551213:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564242816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828195.551220:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828195.551223:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828195.551224:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828195.551227:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3135 00000100:00080000:2.0:1787828195.551229:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828195.551232:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828195.551236:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828195.551242:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564242944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828195.551249:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828195.551252:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828195.551253:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828195.551255:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3135 00000800:00000100:1.0:1787828195.551259:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828195.551260:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828195.551266:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828195.551269:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828195.551277:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564243072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828195.551308:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828195/real 1787828195] req@ffff976889cfa580 x1874670564242560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828195.551323:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564242560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828195.551333:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828195.551337:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828195.551340:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564242560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828195.551361:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564242688:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828195.551377:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828195.551387:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828195/real 1787828195] req@ffff976889cfad00 x1874670564242688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828195.551398:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564242688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828195.551416:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828195.551421:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828195.551425:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564242688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828195.551433:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564242816:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828195.551454:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828195.551463:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828195/real 1787828195] req@ffff976889cf92c0 x1874670564242816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828195.551471:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564242816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828195.551477:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828195.551479:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828195.551482:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564242816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828195.551485:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564242944:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828195.551498:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828195.551505:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828195/real 1787828195] req@ffff976889cf9e00 x1874670564242944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828195.551512:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564242944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828195.551517:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828195.551519:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828195.551521:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564242944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828195.551524:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564243072:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828195.551535:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828195.551541:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828195/real 1787828195] req@ffff976889cfb480 x1874670564243072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828195.551548:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564243072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828195.551553:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828195.551555:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828195.551557:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564243072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828195.806993:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828195.806994:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828195.807089:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828196.574919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828196.830920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828196.830921:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828196.830946:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828197.598929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828197.854942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828197.854942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828197.855006:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828198.622911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828198.878912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828198.878917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828198.878934:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828199.646907:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828199.902927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828199.902927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828199.902984:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828200.670927:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828200.670938:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828200.670943:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828200.670954:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3140 00000100:00080000:1.0:1787828200.670965:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828200.670973:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828200.670978:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828200.671015:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564243200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828200.671045:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828200.671050:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828200.671053:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828200.671059:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3140 00000100:00080000:1.0:1787828200.671065:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828200.671071:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828200.671078:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564243200:192.168.203.139@tcp:250:kworker.0 00000008:00100000:1.0:1787828200.671081:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828200.671110:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564243328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828200.671130:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828200.671134:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828200.671138:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828200.671145:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3140 00000100:00080000:1.0:1787828200.671151:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828200.671157:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:0.0:1787828200.671164:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787828200.671165:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828200.671183:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564243456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828200.671197:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828200.671202:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828200.671205:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828200.671214:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3140 00000100:00080000:1.0:1787828200.671220:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828200.671224:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828200/real 1787828200] req@ffff976889efc780 x1874670564243200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828200.671231:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828200.671248:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828200.671256:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564243200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828200.671268:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564243584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828200.671272:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828200.671280:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828200.671290:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564243200:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787828200.671308:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828200.671314:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828200.671317:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828200.671322:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3140 00000100:00080000:1.0:1787828200.671327:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828200.671332:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828200.671337:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828200.671340:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564243328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828200.671353:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564243712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828200.671373:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828200.671390:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828200/real 1787828200] req@ffff976889eff480 x1874670564243328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828200.671400:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564243328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828200.671407:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828200.671410:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828200.671412:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564243328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828200.671425:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564243456:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828200.671442:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828200.671451:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828200/real 1787828200] req@ffff976889efe580 x1874670564243456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828200.671460:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564243456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828200.671472:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828200.671475:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828200.671478:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564243456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828200.671483:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564243584:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828200.671498:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828200.671509:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828200/real 1787828200] req@ffff976889efe1c0 x1874670564243584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828200.671519:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564243584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828200.671526:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828200.671532:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828200.671535:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564243584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828200.671564:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564243712:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828200.671583:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828200.671599:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828200/real 1787828200] req@ffff976889efcb40 x1874670564243712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828200.671609:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564243712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828200.671615:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828200.671618:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828200.671621:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564243712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828200.926947:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828200.926952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828200.926965:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828201.694937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828201.950923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828201.950948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828201.950980:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828202.718924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828202.974989:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828202.974990:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828202.975020:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828203.742953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828203.998962:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828203.998962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828203.999023:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828204.766925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828205.022927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828205.022929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828205.023097:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828205.790912:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828205.790917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828205.790923:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828205.790928:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828205.790938:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3145 00000100:00080000:1.0:1787828205.790946:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828205.790952:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828205.790986:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564243840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828205.791008:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828205.791011:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828205.791012:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828205.791015:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3145 00000100:00080000:1.0:1787828205.791018:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828205.791021:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828205.791030:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828205.791042:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564243840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828205.791042:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564243968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828205.791057:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828205.791060:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828205.791061:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828205.791064:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3145 00000100:00080000:1.0:1787828205.791067:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828205.791071:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828205.791076:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828205.791090:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564244096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828205.791101:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828205.791106:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787828205.791108:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828205.791112:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828205.791119:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3145 00000100:00080000:1.0:1787828205.791125:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828205.791128:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828205.791133:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828205.791144:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828205/real 1787828205] req@ffff976889efe940 x1874670564243840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828205.791146:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564244224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828205.791177:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564243840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828205.791184:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828205.791191:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828205.791191:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828205.791197:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:1.0:1787828205.791197:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828205.791204:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564243840:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787828205.791207:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3145 00000100:00080000:1.0:1787828205.791212:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828205.791216:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828205.791220:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828205.791228:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564243968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828205.791233:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564244352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828205.791250:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828205.791262:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828205/real 1787828205] req@ffff976889eff0c0 x1874670564243968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828205.791270:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564243968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828205.791277:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828205.791279:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828205.791281:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564243968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828205.791285:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564244096:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828205.791296:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828205.791302:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828205/real 1787828205] req@ffff976889effc00 x1874670564244096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828205.791309:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564244096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828205.791314:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828205.791316:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828205.791318:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564244096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828205.791322:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564244224:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828205.791332:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828205.791338:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828205/real 1787828205] req@ffff976889efed00 x1874670564244224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828205.791345:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564244224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828205.791350:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828205.791352:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828205.791354:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564244224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828205.791365:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564244352:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828205.791375:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828205.791381:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828205/real 1787828205] req@ffff976889efc3c0 x1874670564244352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828205.791392:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564244352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828205.791401:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828205.791404:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828205.791408:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564244352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828206.046922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828206.046922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828206.046980:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828206.814909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828207.070898:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828207.070902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828207.070902:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828207.838935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828208.094939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828208.094940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828208.095007:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828208.862930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828209.118905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828209.118905:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828209.118938:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828209.886926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828210.142872:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828210.142887:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828210.142931:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828210.910902:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828210.910904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828210.910913:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828210.910918:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828210.910928:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3150 00000100:00080000:1.0:1787828210.910939:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828210.910947:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828210.910983:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564244480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828210.911010:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828210.911014:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828210.911017:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828210.911022:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3150 00000100:00080000:1.0:1787828210.911026:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828210.911030:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828210.911039:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828210.911047:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564244480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828210.911054:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564244608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828210.911066:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828210.911070:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828210.911072:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828210.911076:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3150 00000100:00080000:1.0:1787828210.911091:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828210.911096:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:0.0:1787828210.911100:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787828210.911102:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828210.911116:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564244736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828210.911127:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828210.911130:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828210.911132:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828210.911137:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828210/real 1787828210] req@ffff976889efc3c0 x1874670564244480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828210.911138:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3150 00000100:00080000:1.0:1787828210.911145:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828210.911151:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828210.911160:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828210.911168:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564244480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828210.911174:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564244864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828210.911187:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828210.911191:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828210.911196:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:1.0:1787828210.911197:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828210.911202:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828210.911206:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564244480:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787828210.911209:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3150 00000100:00080000:1.0:1787828210.911216:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828210.911220:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828210.911224:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828210.911234:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564244992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828210.911236:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564244608:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828210.911272:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828210.911286:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828210/real 1787828210] req@ffff976889efed00 x1874670564244608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828210.911295:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564244608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828210.911301:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828210.911304:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828210.911306:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564244608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828210.911310:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564244736:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828210.911324:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828210.911330:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828210/real 1787828210] req@ffff976889effc00 x1874670564244736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828210.911337:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564244736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828210.911347:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828210.911348:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828210.911350:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564244736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828210.911354:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564244864:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828210.911364:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828210.911369:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828210/real 1787828210] req@ffff976889eff0c0 x1874670564244864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828210.911376:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564244864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828210.911388:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828210.911391:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828210.911393:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564244864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828210.911407:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564244992:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828210.911418:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828210.911424:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828210/real 1787828210] req@ffff976889efe940 x1874670564244992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828210.911432:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564244992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828210.911437:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828210.911439:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828210.911441:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564244992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828211.166909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828211.166910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828211.166933:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828211.934954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828212.190960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828212.190970:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828212.191035:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828212.958975:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828213.214910:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828213.214945:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828213.214952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828213.982898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828214.238895:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828214.238895:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828214.238955:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828215.006924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828215.262942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828215.262943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828215.262966:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828216.030900:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828216.030908:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1787828216.030909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828216.030912:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828216.030922:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3155 00000100:00080000:1.0:1787828216.030931:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828216.030937:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828216.030971:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564245120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828216.030995:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828216.030999:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828216.031000:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828216.031004:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3155 00000100:00080000:1.0:1787828216.031007:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828216.031010:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828216.031018:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828216.031028:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564245120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828216.031031:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564245248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828216.031041:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828216.031044:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828216.031045:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828216.031048:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3155 00000100:00080000:1.0:1787828216.031050:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828216.031052:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828216.031056:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828216.031064:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564245376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828216.031069:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828216.031071:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828216.031073:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828216.031076:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3155 00000100:00080000:1.0:1787828216.031078:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828216.031080:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828216.031085:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1787828216.031088:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828216.031096:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564245504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828216.031102:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828216.031104:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828216.031105:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828216.031108:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3155 00000100:00080000:1.0:1787828216.031110:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828216.031113:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828216.031116:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828216.031126:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828216/real 1787828216] req@ffff976889efcb40 x1874670564245120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828216.031127:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564245632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828216.031163:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564245120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828216.031174:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828216.031179:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828216.031183:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564245120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828216.031205:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564245248:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828216.031229:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828216.031241:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828216/real 1787828216] req@ffff976889efe1c0 x1874670564245248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828216.031250:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564245248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828216.031257:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828216.031259:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828216.031261:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564245248:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828216.031266:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564245376:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828216.031279:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828216.031287:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828216/real 1787828216] req@ffff976889efe580 x1874670564245376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828216.031299:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564245376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828216.031308:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828216.031312:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828216.031315:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564245376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828216.031321:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564245504:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828216.031345:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828216.031356:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828216/real 1787828216] req@ffff976889eff480 x1874670564245504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828216.031365:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564245504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828216.031372:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828216.031375:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828216.031379:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564245504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828216.031385:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564245632:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828216.031406:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828216.031417:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828216/real 1787828216] req@ffff976889efc780 x1874670564245632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828216.031429:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564245632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828216.031435:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828216.031437:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828216.031439:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564245632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828216.286910:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828216.286931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828216.286939:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828217.054951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828217.310940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828217.310940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828217.310976:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828218.078911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828218.334891:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828218.334894:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828218.334943:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828219.102905:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828219.358890:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828219.358891:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828219.358929:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828220.126915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828220.382908:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828220.382911:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828220.382952:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828221.150927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828221.150928:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828221.150937:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828221.150940:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828221.150949:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3161 00000100:00080000:2.0:1787828221.150958:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828221.150964:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828221.150995:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564245760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828221.151016:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828221.151019:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828221.151020:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828221.151023:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3161 00000100:00080000:2.0:1787828221.151052:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828221.151059:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828221.151072:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828221.151087:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564245760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828221.151097:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564245888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828221.151116:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828221.151124:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828221.151128:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828221.151138:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3161 00000100:00080000:2.0:1787828221.151148:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828221.151172:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828221.151182:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828221.151195:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828221.151208:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564246016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828221.151228:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828221.151233:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828221.151236:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828221.151242:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3161 00000100:00080000:2.0:1787828221.151249:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828221.151251:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828221/real 1787828221] req@ffff976889cfa1c0 x1874670564245760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828221.151258:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828221.151283:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828221.151298:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564246144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828221.151303:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564245760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828221.151308:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828221.151314:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828221.151317:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828221.151322:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828221.151322:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3161 00000100:00080000:2.0:1787828221.151329:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828221.151330:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828221.151334:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564245760:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828221.151340:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828221.151344:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828221.151352:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564246272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828221.151356:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564245888:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828221.151391:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828221.151400:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828221/real 1787828221] req@ffff976889cfb480 x1874670564245888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828221.151408:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564245888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828221.151414:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828221.151416:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828221.151419:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564245888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828221.151422:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564246016:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828221.151434:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828221.151440:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828221/real 1787828221] req@ffff976889cf9e00 x1874670564246016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828221.151447:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564246016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828221.151457:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828221.151459:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828221.151461:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564246016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828221.151465:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564246144:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828221.151475:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828221.151480:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828221/real 1787828221] req@ffff976889cf92c0 x1874670564246144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828221.151487:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564246144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828221.151492:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828221.151494:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828221.151496:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564246144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828221.151506:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564246272:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828221.151516:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828221.151522:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828221/real 1787828221] req@ffff976889cfad00 x1874670564246272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828221.151529:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564246272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828221.151534:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828221.151536:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828221.151538:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564246272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828221.406918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828221.406918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828221.406946:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828222.174920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828222.430933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828222.430933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828222.430949:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828223.198904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828223.454937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828223.454937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828223.454971:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828224.222900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828224.478888:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828224.478905:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828224.478906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828225.246888:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828225.502880:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828225.502902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828225.502923:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828226.270865:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828226.270914:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828226.270920:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828226.270925:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828226.270934:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3166 00000100:00080000:1.0:1787828226.270945:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828226.270952:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828226.270986:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564246400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828226.271011:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828226.271014:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828226.271016:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828226.271021:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3166 00000100:00080000:1.0:1787828226.271024:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828226.271030:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828226.271040:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828226.271041:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564246400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828226.271057:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564246528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828226.271069:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828226.271073:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828226.271074:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828226.271078:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3166 00000100:00080000:1.0:1787828226.271090:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828226.271094:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:0.0:1787828226.271099:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787828226.271102:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828226.271116:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564246656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828226.271128:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828226.271132:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787828226.271135:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828226/real 1787828226] req@ffff976889efc780 x1874670564246400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828226.271136:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828226.271147:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3166 00000100:00080000:1.0:1787828226.271153:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828226.271157:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564246400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828226.271161:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828226.271171:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00080000:0.0:1787828226.271174:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828226.271179:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828226.271183:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564246400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828226.271187:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564246784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828226.271199:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828226.271202:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787828226.271205:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564246528:192.168.203.139@tcp:38:kworker.0 00000100:00080000:1.0:1787828226.271206:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828226.271215:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3166 00000100:00080000:1.0:1787828226.271219:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828226.271224:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:0.0:1787828226.271228:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787828226.271229:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828226.271240:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828226/real 1787828226] req@ffff976889eff480 x1874670564246528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828226.271246:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564246912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828226.271256:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564246528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828226.271268:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828226.271271:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828226.271274:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564246528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828226.271279:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564246656:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828226.271292:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828226.271298:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828226/real 1787828226] req@ffff976889efe580 x1874670564246656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828226.271306:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564246656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828226.271311:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828226.271313:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828226.271315:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564246656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828226.271318:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564246784:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828226.271328:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828226.271333:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828226/real 1787828226] req@ffff976889efe1c0 x1874670564246784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828226.271340:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564246784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828226.271344:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828226.271346:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828226.271349:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564246784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828226.271360:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564246912:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828226.271369:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828226.271375:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828226/real 1787828226] req@ffff976889efcb40 x1874670564246912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828226.271382:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564246912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828226.271387:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828226.271388:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828226.271390:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564246912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828226.526884:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828226.526894:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828226.526930:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828227.294904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828227.550928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828227.550933:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828227.551040:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828228.318881:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828228.574898:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828228.574907:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828228.574923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828229.342897:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828229.598922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828229.598923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828229.598926:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828230.366921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828230.622918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828230.622946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828230.622979:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828231.390916:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828231.390928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828231.390929:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828231.390937:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828231.390950:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3171 00000100:00080000:1.0:1787828231.390961:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828231.390969:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828231.391006:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564247040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828231.391031:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828231.391034:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828231.391035:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828231.391038:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3171 00000100:00080000:1.0:1787828231.391041:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828231.391044:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828231.391052:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828231.391061:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564247168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828231.391070:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828231.391084:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787828231.391085:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564247040:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787828231.391087:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828231.391098:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3171 00000100:00080000:1.0:1787828231.391101:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828231.391104:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828231.391107:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828231.391118:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564247296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828231.391130:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828231.391132:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828231.391133:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828231.391136:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3171 00000100:00080000:1.0:1787828231.391138:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828231.391141:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828231.391144:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828231.391152:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564247424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828231.391158:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828231.391161:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828231.391163:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828231.391165:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3171 00000100:00080000:1.0:1787828231.391167:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828231.391170:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828231.391173:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828231.391178:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828231.391187:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564247552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828231.391232:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828231/real 1787828231] req@ffff976889efe940 x1874670564247040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828231.391254:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564247040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828231.391264:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828231.391268:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828231.391271:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564247040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828231.391295:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564247168:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828231.391312:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828231.391321:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828231/real 1787828231] req@ffff976889eff0c0 x1874670564247168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828231.391330:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564247168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828231.391336:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828231.391338:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828231.391340:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564247168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828231.391344:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564247296:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828231.391354:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828231.391360:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828231/real 1787828231] req@ffff976889effc00 x1874670564247296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828231.391368:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564247296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828231.391378:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828231.391380:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828231.391382:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564247296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828231.391386:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564247424:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828231.391397:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828231.391403:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828231/real 1787828231] req@ffff976889efed00 x1874670564247424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828231.391410:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564247424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828231.391416:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828231.391418:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828231.391420:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564247424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828231.391425:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564247552:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828231.391435:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828231.391440:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828231/real 1787828231] req@ffff976889efc3c0 x1874670564247552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828231.391447:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564247552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828231.391453:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828231.391454:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828231.391457:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564247552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828231.646902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828231.646939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828231.646978:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828232.414941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828232.670900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828232.670915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828232.670942:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828233.438911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828233.694961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828233.694977:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828233.695069:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828234.462976:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828234.718920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828234.718920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828234.718958:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828235.486938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828235.742958:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828235.742990:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828235.743123:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828236.510938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828236.510943:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828236.510950:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828236.510956:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828236.510974:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3176 00000100:00080000:3.0:1787828236.510990:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828236.510998:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828236.511049:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564247680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828236.511083:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828236.511086:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828236.511088:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828236.511092:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3176 00000100:00080000:3.0:1787828236.511096:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828236.511101:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828236.511112:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828236.511122:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564247680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828236.511130:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564247808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828236.511141:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828236.511144:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828236.511146:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828236.511149:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3176 00000100:00080000:3.0:1787828236.511154:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828236.511156:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828236.511161:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828236.511169:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564247936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828236.511175:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828236.511178:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828236.511179:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828236.511184:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3176 00000100:00080000:3.0:1787828236.511188:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828236.511192:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828236.511196:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828236.511204:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564248064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828236.511211:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828236.511214:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828236.511216:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828236.511222:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787828236.511222:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3176 00000100:00080000:3.0:1787828236.511226:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828236.511231:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828236.511235:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828236.511245:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564248192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828236.511288:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828236/real 1787828236] req@ffff97688cb5c3c0 x1874670564247680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828236.511305:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564247680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828236.511314:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828236.511318:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828236.511321:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564247680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828236.511368:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564247808:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828236.511400:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828236.511422:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828236/real 1787828236] req@ffff97688cb5cf00 x1874670564247808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828236.511441:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564247808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828236.511477:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828236.511484:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828236.511491:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564247808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828236.511504:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564247936:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828236.511538:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828236.511556:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828236/real 1787828236] req@ffff97688cb5c780 x1874670564247936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828236.511590:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564247936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828236.511627:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828236.511632:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828236.511653:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564247936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828236.511659:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564248064:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828236.511682:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828236.511697:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828236/real 1787828236] req@ffff97688cb5e1c0 x1874670564248064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828236.511712:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564248064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828236.511723:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828236.511727:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828236.511734:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564248064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828236.511740:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564248192:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828236.511758:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828236.511769:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828236/real 1787828236] req@ffff97688cb5d2c0 x1874670564248192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828236.511781:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564248192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828236.511789:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828236.511792:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828236.511795:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564248192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828236.766913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828236.766934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828236.766962:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828237.534969:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828237.790918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828237.790984:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828237.790989:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828238.558999:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828238.815118:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828238.815118:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828238.815234:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828239.582936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828239.838972:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828239.838978:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828239.839065:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828240.606928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828240.863042:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828240.863052:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828240.863183:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828241.630919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828241.630923:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828241.630934:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828241.630938:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828241.630949:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3181 00000100:00080000:3.0:1787828241.630961:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828241.630969:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828241.631024:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564248320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828241.631052:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828241.631055:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828241.631057:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828241.631060:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3181 00000100:00080000:3.0:1787828241.631063:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828241.631066:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828241.631075:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828241.631085:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564248448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828241.631094:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828241.631098:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828241.631099:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828241.631103:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3181 00000100:00080000:3.0:1787828241.631106:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828241.631109:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564248320:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787828241.631111:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828241.631122:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828241.631136:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564248576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828241.631158:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828241.631162:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828241.631165:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828241.631170:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3181 00000100:00080000:3.0:1787828241.631176:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828241.631182:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828241.631189:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828241.631205:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564248704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828241.631215:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564248448:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787828241.631228:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828241.631237:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828241.631240:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828241.631248:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3181 00000100:00080000:3.0:1787828241.631254:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828241.631259:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828241.631266:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828241.631267:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564248576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828241.631308:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564248832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828241.631329:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564248704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828241.631402:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564248832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828241.886941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828241.886942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828241.887045:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828242.654952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828242.911019:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828242.911019:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828242.911093:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828243.678911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828243.934937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828243.934937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828243.935058:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828244.702985:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:0.0:1787828244.703343:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:0.0:1787828244.703409:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828244.703434:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828244.703498:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828244.703510:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828244.703525:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828244.703538:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:1.0:1787828244.703577:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828241/real 1787828244] req@ffff97688cb5f0c0 x1874670564248576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828244.703618:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564248576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828244.703638:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828244.703647:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828244.703653:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564248576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828244.703664:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828241/real 1787828244] req@ffff97688cb5d680 x1874670564248704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828244.703679:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564248704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828244.703689:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828244.703695:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828244.703699:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564248704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828244.703709:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828241/real 1787828244] req@ffff97688cb5f840 x1874670564248320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828244.703725:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564248320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828244.703732:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828244.703735:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828244.703737:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564248320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828244.703742:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828241/real 1787828244] req@ffff97688cb5de00 x1874670564248448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828244.703750:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564248448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828244.703756:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828244.703760:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828244.703763:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564248448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828244.703817:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828241/real 1787828244] req@ffff97688cb5cb40 x1874670564248832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828244.703867:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564248832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828244.703877:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828244.703890:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828244.703893:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564248832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828244.958962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828244.958962:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828244.958969:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828245.726946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828245.982939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828245.983005:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828245.983050:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828246.750964:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828246.751111:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828246.751125:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828246.751133:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828246.751151:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3186 00000100:00080000:2.0:1787828246.751168:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828246.751182:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828246.751236:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564248960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828246.751298:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828246.751305:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828246.751309:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828246.751329:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3186 00000100:00080000:2.0:1787828246.751337:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828246.751346:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828246.751364:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828246.751392:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564248960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828246.751401:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564249088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828246.751432:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828246.751439:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828246.751443:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828246.751452:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3186 00000100:00080000:2.0:1787828246.751461:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828246.751469:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828246.751480:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828246.751504:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828246.751511:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564249216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828246.751568:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828246.751576:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828246.751580:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828246.751590:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3186 00000100:00080000:2.0:1787828246.751599:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828246.751607:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828246/real 1787828246] req@ffff976889cfa580 x1874670564248960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828246.751614:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828246.751633:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828246.751669:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564248960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828246.751674:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564249344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828246.751720:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828246.751727:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828246.751737:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:2.0:1787828246.751740:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828246.751750:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828246.751754:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564248960:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828246.751768:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3186 00000100:00080000:2.0:1787828246.751780:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828246.751788:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828246.751798:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828246.751812:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564249088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828246.751874:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564249472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828246.752058:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828246.752089:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828246/real 1787828246] req@ffff976889cfa940 x1874670564249088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828246.752117:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564249088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828246.752137:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828246.752144:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828246.752152:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564249088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828246.752166:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564249216:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828246.752209:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828246.752235:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828246/real 1787828246] req@ffff976889cfbc00 x1874670564249216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828246.752260:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564249216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828246.752278:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828246.752284:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828246.752291:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564249216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828246.752305:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564249344:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828246.752350:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828246.752375:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828246/real 1787828246] req@ffff976889cf8000 x1874670564249344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828246.752399:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564249344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828246.752416:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828246.752423:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828246.752429:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564249344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828246.752463:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564249472:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828246.752500:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828246.752523:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828246/real 1787828246] req@ffff976889cfb0c0 x1874670564249472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828246.752578:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564249472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828246.752595:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828246.752601:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828246.752607:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564249472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828247.006951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828247.006952:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828247.006991:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828247.774954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787828248.031007:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828248.031033:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828248.031036:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828248.798944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828249.054941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828249.054988:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828249.055011:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828249.822909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828250.078903:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828250.078917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828250.078941:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828250.846928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828251.102896:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828251.102898:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828251.102944:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828251.870922:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828251.870933:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1787828251.870934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828251.870943:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828251.870954:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3191 00000100:00080000:0.0:1787828251.870964:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828251.870971:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828251.871008:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564249600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828251.871030:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828251.871033:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828251.871036:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828251.871039:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3191 00000100:00080000:0.0:1787828251.871041:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828251.871044:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828251.871052:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828251.871061:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564249728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828251.871068:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828251.871070:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828251.871072:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828251.871074:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3191 00000100:00080000:0.0:1787828251.871076:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828251.871079:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828251.871082:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828251.871089:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564249856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828251.871098:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828251.871100:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828251.871102:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828251.871107:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3191 00000100:00100000:1.0:1787828251.871109:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564249600:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828251.871110:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828251.871117:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828251.871124:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828251.871133:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564249984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828251.871139:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828251.871142:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828251.871143:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828251.871146:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3191 00000100:00080000:0.0:1787828251.871149:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828251.871151:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828251.871155:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828251.871162:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564250112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828251.871178:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828251.871220:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828251/real 1787828251] req@ffff976882aef0c0 x1874670564249600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828251.871234:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564249600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828251.871243:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828251.871247:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828251.871251:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564249600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828251.871258:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564249728:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828251.871277:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828251.871288:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828251/real 1787828251] req@ffff976882aee940 x1874670564249728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828251.871299:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564249728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828251.871305:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828251.871308:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828251.871310:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564249728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828251.871330:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564249856:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828251.871342:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828251.871371:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828251/real 1787828251] req@ffff976882aed680 x1874670564249856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828251.871383:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564249856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828251.871392:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828251.871395:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828251.871398:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564249856:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828251.871404:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564249984:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828251.871422:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828251.871432:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828251/real 1787828251] req@ffff976882aee580 x1874670564249984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828251.871452:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564249984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828251.871460:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828251.871463:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828251.871466:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564249984:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828251.871471:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564250112:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828251.871487:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828251.871539:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828251/real 1787828251] req@ffff976882aec3c0 x1874670564250112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828251.871548:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564250112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828251.871554:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828251.871557:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828251.871559:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564250112:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828252.126891:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828252.126891:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828252.126909:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828252.894879:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828253.150911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828253.150914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828253.150936:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828253.918982:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828254.174933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828254.174933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828254.174958:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828254.942921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828255.198902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828255.198902:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828255.198959:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828255.966959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828256.222922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828256.222922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828256.222962:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828256.990951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828256.990979:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828256.990992:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828256.990998:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828256.991033:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3196 00000100:00080000:1.0:1787828256.991051:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828256.991063:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828256.991123:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564250240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828256.991165:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828256.991171:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828256.991174:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828256.991182:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3196 00000100:00080000:1.0:1787828256.991189:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828256.991197:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828256.991213:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828256.991219:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564250240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828256.991233:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564250368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828256.991248:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828256.991254:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828256.991257:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828256.991263:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3196 00000100:00080000:1.0:1787828256.991271:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828256.991278:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828256.991286:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828256.991306:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564250496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828256.991311:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787828256.991333:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828256.991337:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828256.991340:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828256.991346:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3196 00000100:00080000:1.0:1787828256.991351:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828256.991355:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828256.991362:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828256.991373:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828256/real 1787828256] req@ffff976889efc3c0 x1874670564250240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828256.991379:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564250624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828256.991414:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828256.991418:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828256.991420:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828256.991425:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564250240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828256.991429:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3196 00000100:00080000:1.0:1787828256.991436:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828256.991445:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828256.991453:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828256.991472:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828256.991482:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828256.991489:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564250752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828256.991493:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564250240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828256.991556:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564250368:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828256.991588:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828256.991604:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828256/real 1787828256] req@ffff976889efed00 x1874670564250368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828256.991618:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564250368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828256.991627:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828256.991631:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828256.991634:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564250368:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828256.991641:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564250496:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828256.991659:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828256.991669:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828256/real 1787828256] req@ffff976889effc00 x1874670564250496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828256.991680:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564250496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828256.991689:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828256.991692:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828256.991696:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564250496:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828256.991702:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564250624:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828256.991741:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828256.991760:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828256/real 1787828256] req@ffff976889eff0c0 x1874670564250624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828256.991798:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564250624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828256.991813:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828256.991819:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828256.991824:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564250624:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828256.991834:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564250752:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828256.991878:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828256.991946:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828256/real 1787828256] req@ffff976889efe940 x1874670564250752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828256.991958:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564250752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828256.991967:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828256.991970:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828256.991974:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564250752:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828257.246957:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828257.246973:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828257.247029:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828258.014965:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828258.271004:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828258.271012:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828258.271059:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828259.038952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828259.294882:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828259.294933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828259.294955:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828260.062922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828260.318946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828260.318946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828260.318984:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828261.087020:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787828261.342995:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828261.343027:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828261.343036:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00080000:2.0:1787828262.110985:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828262.110989:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828262.110998:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828262.111003:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828262.111015:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3202 00000100:00080000:2.0:1787828262.111027:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828262.111035:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828262.111082:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564250880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828262.111107:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828262.111109:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828262.111111:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828262.111126:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3202 00000100:00080000:2.0:1787828262.111129:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828262.111132:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828262.111142:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828262.111154:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564251008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828262.111161:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564250880:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828262.111165:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828262.111171:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828262.111173:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828262.111178:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3202 00000100:00080000:2.0:1787828262.111182:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828262.111186:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828262.111192:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828262.111216:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564251136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828262.111228:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828262.111230:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828262.111232:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828262.111236:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3202 00000100:00080000:2.0:1787828262.111239:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828262.111243:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828262.111248:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828262.111277:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564251264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828262.111290:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828262.111296:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828262.111302:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828262.111305:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828262.111310:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3202 00000100:00080000:2.0:1787828262.111315:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828262.111336:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828262.111341:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828262.111352:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564251392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828262.111405:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828262/real 1787828262] req@ffff976889cf8f00 x1874670564250880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828262.111458:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564250880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828262.111476:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828262.111484:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828262.111492:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564250880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828262.111529:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564251008:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828262.111571:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828262.111594:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828262/real 1787828262] req@ffff976889cfb0c0 x1874670564251008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828262.111614:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564251008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828262.111628:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828262.111633:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828262.111639:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564251008:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828262.111649:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564251136:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828262.111689:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828262.111708:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828262/real 1787828262] req@ffff976889cf8000 x1874670564251136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828262.111726:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564251136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828262.111749:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828262.111755:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828262.111761:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564251136:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828262.111770:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564251264:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828262.111799:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828262.111816:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828262/real 1787828262] req@ffff976889cfbc00 x1874670564251264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828262.111912:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564251264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828262.111925:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828262.111931:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828262.111937:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564251264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828262.111949:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564251392:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828262.112024:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828262.112037:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828262/real 1787828262] req@ffff976889cfa940 x1874670564251392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828262.112051:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564251392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828262.112059:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828262.112062:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828262.112066:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564251392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828262.366915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828262.366915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828262.366950:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828263.134891:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828263.391002:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828263.391003:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828263.391023:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828264.158967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828264.414913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828264.414924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828264.414972:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828265.182999:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828265.438975:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828265.438975:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828265.439032:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828266.206941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828266.462931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828266.462945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828266.462990:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828267.230895:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828267.230927:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828267.230933:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828267.230938:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828267.230948:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3207 00000100:00080000:3.0:1787828267.230960:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828267.230967:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828267.231017:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564251520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828267.231041:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828267.231044:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828267.231046:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828267.231052:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3207 00000100:00080000:3.0:1787828267.231055:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828267.231058:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828267.231068:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828267.231080:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564251648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828267.231090:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828267.231095:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564251520:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787828267.231095:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828267.231097:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828267.231105:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3207 00000100:00080000:3.0:1787828267.231108:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828267.231110:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828267.231114:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828267.231122:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564251776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828267.231128:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828267.231131:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828267.231132:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828267.231135:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3207 00000100:00080000:3.0:1787828267.231137:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828267.231139:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828267.231142:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828267.231149:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564251904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828267.231155:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828267.231157:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828267.231159:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828267.231161:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3207 00000100:00080000:3.0:1787828267.231164:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828267.231167:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828267.231171:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828267.231178:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564252032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828267.231186:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828267.231243:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828267/real 1787828267] req@ffff97688cb5c000 x1874670564251520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828267.231258:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564251520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828267.231267:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828267.231271:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828267.231274:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564251520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828267.231298:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564251648:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828267.231315:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828267.231324:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828267/real 1787828267] req@ffff97688cb5cb40 x1874670564251648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828267.231333:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564251648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828267.231339:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828267.231341:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828267.231344:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564251648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828267.231348:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564251776:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828267.231359:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828267.231366:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828267/real 1787828267] req@ffff97688cb5de00 x1874670564251776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828267.231373:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564251776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828267.231378:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828267.231380:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828267.231382:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564251776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828267.231400:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564251904:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828267.231415:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828267.231423:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828267/real 1787828267] req@ffff97688cb5f840 x1874670564251904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828267.231431:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564251904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828267.231436:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828267.231439:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828267.231441:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564251904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828267.231444:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564252032:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828267.231455:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828267.231461:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828267/real 1787828267] req@ffff97688cb5d680 x1874670564252032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828267.231468:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564252032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828267.231474:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828267.231475:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828267.231477:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564252032:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828267.486951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828267.486952:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828267.486985:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828268.254950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828268.511006:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828268.511077:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828268.511096:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828269.278975:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828269.534987:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828269.534987:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828269.535145:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828270.303016:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828270.558954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828270.558954:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828270.559020:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828271.327015:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828271.582910:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828271.582910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828271.583025:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828272.350966:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828272.350980:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828272.350986:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828272.351000:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3212 00000100:00080000:3.0:1787828272.351014:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828272.351043:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828272.351043:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828272.351101:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564252160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828272.351160:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828272.351165:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828272.351169:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828272.351176:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3212 00000100:00080000:3.0:1787828272.351182:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828272.351188:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828272.351201:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828272.351211:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564252160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828272.351224:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564252288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828272.351260:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828272.351265:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828272.351268:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828272.351275:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3212 00000100:00080000:3.0:1787828272.351282:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828272.351288:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828272.351296:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828272.351316:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564252416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828272.351330:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828272.351342:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787828272.351343:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828272.351379:0:8:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828272.351390:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3212 00000100:00080000:3.0:1787828272.351398:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828272.351404:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828272.351413:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828272.351438:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564252544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828272.351450:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828272/real 1787828272] req@ffff97688cb5f0c0 x1874670564252160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828272.351459:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828272.351500:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564252160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828272.351519:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828272.351525:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828272.351533:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564252160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828272.351578:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564252288:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787828272.351583:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828272.351591:0:8:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828272.351599:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3212 00000100:00080000:3.0:1787828272.351606:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828272.351611:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828272.351640:0:8:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828272.351655:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828272.351692:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828272/real 1787828272] req@ffff97688cb5d2c0 x1874670564252288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828272.351723:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564252288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828272.351736:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828272.351740:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828272.351762:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564252288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828272.351771:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564252416:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828272.351806:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828272.351862:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828272/real 1787828272] req@ffff97688cb5e1c0 x1874670564252416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828272.351884:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564252416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828272.351892:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564252672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828272.351929:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828272.351937:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828272.351945:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564252416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828272.351956:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564252544:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828272.351993:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828272.352025:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828272/real 1787828272] req@ffff97688cb5c780 x1874670564252544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828272.352048:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564252544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828272.352065:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828272.352069:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828272.352073:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564252544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828272.352099:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564252672:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828272.352118:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828272.352130:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828272/real 1787828272] req@ffff97688cb5cf00 x1874670564252672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828272.352142:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564252672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828272.352151:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828272.352155:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828272.352159:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564252672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828272.606932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828272.606964:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828272.606997:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828273.374920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828273.631000:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828273.631054:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828273.631100:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828274.398954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828274.654922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828274.654926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828274.655080:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828275.422959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828275.678972:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828275.678973:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828275.679096:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828276.446936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828276.702900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828276.702900:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828276.703006:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828277.470933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828277.470972:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828277.470978:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828277.470981:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828277.470990:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3217 00000100:00080000:0.0:1787828277.470999:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828277.471004:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828277.471043:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564252800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828277.471063:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828277.471066:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828277.471068:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828277.471071:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3217 00000100:00080000:0.0:1787828277.471073:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828277.471076:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828277.471084:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828277.471094:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564252928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828277.471098:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564252800:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828277.471116:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828277.471119:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828277.471120:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828277.471123:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3217 00000100:00080000:0.0:1787828277.471125:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828277.471127:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828277.471130:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828277.471139:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564253056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828277.471146:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828277.471149:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828277.471150:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828277.471152:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3217 00000100:00080000:0.0:1787828277.471157:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828277.471159:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828277.471164:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1787828277.471167:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828277.471176:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564253184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828277.471184:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828277.471187:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828277.471188:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828277.471190:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3217 00000100:00080000:0.0:1787828277.471193:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828277.471196:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828277.471199:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828277.471204:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828277/real 1787828277] req@ffff976882aeed00 x1874670564252800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828277.471209:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564253312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828277.471235:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564252800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828277.471245:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828277.471248:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828277.471251:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564252800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828277.471269:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564252928:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828277.471284:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828277.471292:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828277/real 1787828277] req@ffff976882aec3c0 x1874670564252928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828277.471300:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564252928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828277.471306:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828277.471309:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828277.471311:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564252928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828277.471330:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564253056:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828277.471345:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828277.471352:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828277/real 1787828277] req@ffff976882aee580 x1874670564253056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828277.471364:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564253056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828277.471370:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828277.471372:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828277.471374:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564253056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828277.471378:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564253184:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828277.471388:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828277.471394:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828277/real 1787828277] req@ffff976882aed680 x1874670564253184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828277.471401:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564253184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828277.471406:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828277.471408:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828277.471410:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564253184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828277.471413:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564253312:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828277.471422:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828277.471428:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828277/real 1787828277] req@ffff976882aee940 x1874670564253312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828277.471435:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564253312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828277.471444:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828277.471446:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828277.471448:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564253312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828277.726935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828277.726958:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828277.726969:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828278.494894:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828278.750915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828278.750915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828278.750930:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828279.518982:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828279.774905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828279.774906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828279.774961:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828280.542904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828280.798906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828280.798910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828280.798942:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828281.566889:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828281.822926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828281.822926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828281.822981:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828282.590910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828282.590950:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828282.590955:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828282.590958:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828282.590966:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3222 00000100:00080000:2.0:1787828282.590976:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828282.590981:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828282.591013:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564253440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828282.591035:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828282.591038:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828282.591039:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828282.591042:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3222 00000100:00080000:2.0:1787828282.591044:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828282.591047:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828282.591055:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828282.591068:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564253568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828282.591070:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564253440:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828282.591089:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828282.591092:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828282.591093:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828282.591096:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3222 00000100:00080000:2.0:1787828282.591099:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828282.591102:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828282.591105:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828282.591113:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564253696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828282.591119:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828282.591122:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828282.591123:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828282.591126:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3222 00000100:00080000:2.0:1787828282.591128:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828282.591130:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828282.591133:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1787828282.591137:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828282.591144:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564253824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828282.591153:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828282.591155:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828282.591157:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828282.591159:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3222 00000100:00080000:2.0:1787828282.591161:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828282.591163:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828282.591166:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828282.591175:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564253952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828282.591179:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828282/real 1787828282] req@ffff976889cfa580 x1874670564253440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828282.591201:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564253440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828282.591209:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828282.591212:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828282.591215:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564253440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828282.591234:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564253568:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828282.591249:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828282.591258:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828282/real 1787828282] req@ffff976889cfad00 x1874670564253568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828282.591266:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564253568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828282.591272:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828282.591274:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828282.591294:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564253568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828282.591299:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564253696:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828282.591314:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828282.591322:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828282/real 1787828282] req@ffff976889cf92c0 x1874670564253696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828282.591329:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564253696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828282.591335:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828282.591337:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828282.591339:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564253696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828282.591343:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564253824:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828282.591353:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828282.591358:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828282/real 1787828282] req@ffff976889cf9e00 x1874670564253824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828282.591365:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564253824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828282.591371:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828282.591373:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828282.591375:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564253824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828282.591378:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564253952:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828282.591388:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828282.591393:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828282/real 1787828282] req@ffff976889cfb480 x1874670564253952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828282.591399:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564253952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828282.591404:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828282.591406:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828282.591408:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564253952:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828282.846954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828282.846954:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828282.846974:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828283.614913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828283.870911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828283.870914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828283.870972:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828284.638948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828284.894895:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828284.894918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828284.894929:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828285.662916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828285.918906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828285.918906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828285.918951:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828286.686899:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828286.942880:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828286.942896:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828286.942936:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828287.710897:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828287.710943:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828287.710948:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828287.710951:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828287.710959:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3227 00000100:00080000:0.0:1787828287.710968:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828287.710974:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828287.711004:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564254080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828287.711026:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828287.711029:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828287.711030:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828287.711033:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3227 00000100:00080000:0.0:1787828287.711035:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828287.711038:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828287.711046:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828287.711059:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564254080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828287.711060:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564254208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828287.711080:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828287.711082:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828287.711083:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828287.711085:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3227 00000100:00080000:0.0:1787828287.711087:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828287.711089:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828287.711093:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828287.711101:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564254336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828287.711108:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828287.711110:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828287.711112:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828287.711114:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3227 00000100:00080000:0.0:1787828287.711120:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828287.711122:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828287.711126:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1787828287.711130:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828287.711137:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564254464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828287.711145:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828287.711148:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828287.711149:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828287.711151:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3227 00000100:00080000:0.0:1787828287.711153:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828287.711155:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828287.711158:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828287.711166:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564254592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828287.711170:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828287/real 1787828287] req@ffff976882aef0c0 x1874670564254080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828287.711187:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564254080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828287.711196:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828287.711199:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828287.711202:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564254080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828287.711220:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564254208:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828287.711235:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828287.711260:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828287/real 1787828287] req@ffff976882aee940 x1874670564254208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828287.711270:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564254208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828287.711276:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828287.711278:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828287.711280:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564254208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828287.711284:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564254336:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828287.711299:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828287.711306:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828287/real 1787828287] req@ffff976882aed680 x1874670564254336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828287.711318:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564254336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828287.711323:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828287.711325:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828287.711327:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564254336:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828287.711330:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564254464:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828287.711340:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828287.711346:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828287/real 1787828287] req@ffff976882aee580 x1874670564254464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828287.711353:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564254464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828287.711358:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828287.711360:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828287.711362:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564254464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828287.711365:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564254592:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828287.711375:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828287.711380:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828287/real 1787828287] req@ffff976882aec3c0 x1874670564254592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828287.711387:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564254592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828287.711397:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828287.711401:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828287.711404:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564254592:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828287.966900:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828287.966908:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828287.966953:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828288.734907:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828288.990943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828288.990946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828288.990956:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828289.758896:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828290.014929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828290.014929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828290.014987:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828290.782921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828291.038871:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828291.038887:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828291.038918:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828291.806901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828292.062878:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828292.062900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828292.062938:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828292.830919:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828292.830926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828292.830930:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828292.830935:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828292.830945:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3232 00000100:00080000:0.0:1787828292.830954:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828292.830960:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828292.830996:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564254720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828292.831026:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828292.831030:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828292.831032:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828292.831037:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3232 00000100:00080000:0.0:1787828292.831041:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828292.831046:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828292.831056:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828292.831068:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564254848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828292.831076:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828292.831079:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828292.831080:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828292.831084:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3232 00000100:00080000:0.0:1787828292.831087:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828292.831090:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828292.831093:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828292.831099:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564254976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828292.831112:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828292.831114:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564254720:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828292.831115:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828292.831119:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828292.831127:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3232 00000100:00080000:0.0:1787828292.831129:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828292.831131:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828292.831135:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828292.831144:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564255104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828292.831150:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828292.831153:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828292.831154:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828292.831157:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3232 00000100:00080000:0.0:1787828292.831159:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828292.831161:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828292.831164:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828292.831170:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564255232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828292.831191:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828292.831244:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828292/real 1787828292] req@ffff976882aeed00 x1874670564254720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828292.831260:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564254720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828292.831269:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828292.831273:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828292.831277:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564254720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828292.831283:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564254848:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828292.831299:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828292.831308:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828292/real 1787828292] req@ffff976882aec3c0 x1874670564254848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828292.831316:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564254848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828292.831322:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828292.831324:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828292.831326:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564254848:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828292.831345:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564254976:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828292.831356:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828292.831363:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828292/real 1787828292] req@ffff976882aee580 x1874670564254976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828292.831370:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564254976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828292.831376:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828292.831378:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828292.831380:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564254976:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828292.831384:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564255104:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828292.831393:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828292.831399:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828292/real 1787828292] req@ffff976882aed680 x1874670564255104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828292.831406:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564255104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828292.831413:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828292.831415:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828292.831417:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564255104:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828292.831420:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564255232:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828292.831430:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828292.831436:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828292/real 1787828292] req@ffff976882aee940 x1874670564255232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828292.831442:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564255232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828292.831447:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828292.831449:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828292.831451:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564255232:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828293.086897:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828293.086903:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828293.086928:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828293.854959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828294.110877:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828294.110907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828294.110972:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828294.878930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828295.134890:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828295.134897:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828295.134923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828295.902921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828296.158883:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828296.158912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828296.158961:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828296.926934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828297.182884:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828297.182891:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828297.182937:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828297.950942:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828297.950950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828297.950951:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828297.950955:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828297.950970:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3237 00000100:00080000:0.0:1787828297.950982:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828297.950990:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828297.951031:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564255360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828297.951054:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828297.951056:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828297.951058:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828297.951061:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3237 00000100:00080000:0.0:1787828297.951063:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828297.951067:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828297.951075:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828297.951090:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564255488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828297.951098:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564255360:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828297.951103:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828297.951118:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828297.951121:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828297.951124:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3237 00000100:00080000:0.0:1787828297.951126:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828297.951128:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828297.951132:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828297.951143:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564255616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828297.951149:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828297.951152:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828297.951155:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828297.951158:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3237 00000100:00080000:0.0:1787828297.951160:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828297.951163:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828297.951168:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828297.951193:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564255744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828297.951200:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828297.951202:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828297.951204:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828297.951206:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3237 00000100:00080000:0.0:1787828297.951209:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828297.951215:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:1.0:1787828297.951219:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:0.0:1787828297.951225:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828297.951236:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564255872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828297.951280:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828297/real 1787828297] req@ffff976882aef0c0 x1874670564255360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828297.951303:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564255360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828297.951317:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828297.951322:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828297.951326:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564255360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828297.951352:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564255488:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828297.951381:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828297.951397:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828297/real 1787828297] req@ffff976882aee940 x1874670564255488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828297.951411:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564255488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828297.951417:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828297.951420:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828297.951422:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564255488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828297.951428:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564255616:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828297.951443:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828297.951458:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828297/real 1787828297] req@ffff976882aed680 x1874670564255616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828297.951472:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564255616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828297.951479:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828297.951482:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828297.951484:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564255616:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828297.951488:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564255744:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828297.951500:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828297.951506:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828297/real 1787828297] req@ffff976882aee580 x1874670564255744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828297.951515:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564255744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828297.951521:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828297.951523:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828297.951525:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564255744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828297.951529:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564255872:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828297.951540:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828297.951546:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828297/real 1787828297] req@ffff976882aec3c0 x1874670564255872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828297.951552:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564255872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828297.951563:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828297.951567:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828297.951571:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564255872:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828298.206957:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828298.206965:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828298.207008:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828298.974951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828299.230945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828299.230955:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828299.230971:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828299.998929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828300.254897:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828300.254897:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828300.254952:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828301.022938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828301.278927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828301.278927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828301.278963:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828302.046904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828302.302932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828302.302932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828302.303010:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828303.070927:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828303.070928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828303.070937:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828303.070941:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828303.070949:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3242 00000100:00080000:2.0:1787828303.070958:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828303.070965:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828303.070998:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564256000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828303.071022:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828303.071025:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828303.071026:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828303.071029:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3242 00000100:00080000:2.0:1787828303.071031:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828303.071034:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828303.071042:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828303.071052:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564256128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828303.071060:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564256000:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828303.071072:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828303.071074:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828303.071075:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828303.071078:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3242 00000100:00080000:2.0:1787828303.071080:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828303.071082:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828303.071085:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828303.071093:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564256256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828303.071099:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828303.071101:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828303.071102:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828303.071105:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3242 00000100:00080000:2.0:1787828303.071107:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828303.071110:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828303.071113:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828303.071121:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564256384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828303.071126:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828303.071146:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828303.071149:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828303.071150:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828303.071154:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3242 00000100:00080000:2.0:1787828303.071157:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828303.071160:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828303.071164:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828303.071172:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564256512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828303.071185:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828303/real 1787828303] req@ffff976889cfa1c0 x1874670564256000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828303.071200:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564256000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828303.071208:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828303.071211:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828303.071213:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564256000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828303.071232:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564256128:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828303.071248:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828303.071257:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828303/real 1787828303] req@ffff976889cfb480 x1874670564256128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828303.071265:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564256128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828303.071271:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828303.071273:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828303.071276:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564256128:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828303.071279:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564256256:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828303.071289:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828303.071295:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828303/real 1787828303] req@ffff976889cf9e00 x1874670564256256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828303.071302:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564256256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828303.071307:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828303.071309:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828303.071311:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564256256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828303.071314:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564256384:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828303.071324:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828303.071330:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828303/real 1787828303] req@ffff976889cf92c0 x1874670564256384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828303.071337:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564256384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828303.071343:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828303.071345:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828303.071347:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564256384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828303.071350:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564256512:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828303.071359:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828303.071365:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828303/real 1787828303] req@ffff976889cfad00 x1874670564256512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828303.071371:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564256512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828303.071376:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828303.071378:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828303.071380:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564256512:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828303.326939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828303.326974:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828303.327054:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828304.094921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828304.350946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828304.350999:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828304.351053:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828305.118937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828305.374882:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828305.374900:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828305.374941:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828306.142902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828306.398960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828306.399000:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828306.399030:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828307.166933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787828307.422919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828307.422926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828307.422988:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828308.190925:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828308.190931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828308.190935:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828308.190941:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828308.190950:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3248 00000100:00080000:2.0:1787828308.190966:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828308.190972:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828308.191003:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564256640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828308.191025:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828308.191028:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828308.191030:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828308.191033:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3248 00000100:00080000:2.0:1787828308.191036:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828308.191039:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828308.191048:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828308.191059:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564256640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828308.191064:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564256768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828308.191079:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828308.191082:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828308.191083:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828308.191086:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3248 00000100:00080000:2.0:1787828308.191090:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828308.191093:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828308.191097:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828308.191122:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564256896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828308.191131:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828308.191133:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828308.191135:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828308.191138:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3248 00000100:00080000:2.0:1787828308.191143:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828308.191149:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828308.191155:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828308.191168:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564257024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828308.191173:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564256768:192.168.203.139@tcp:38:kworker.0 00000100:00080000:2.0:1787828308.191190:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828308.191194:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828308.191197:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828308.191201:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3248 00000100:00100000:0.0:1787828308.191206:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564256896:192.168.203.139@tcp:38:kworker.0 00000100:00080000:2.0:1787828308.191207:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828308.191215:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828308.191219:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828308.191232:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564257152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828308.191245:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564257024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828308.191261:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564257152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828308.446892:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828308.446939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828308.446972:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828309.215001:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828309.470949:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828309.470971:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828309.470989:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828310.239007:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828310.494912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828310.494933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828310.495044:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828311.263109:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:0.0:1787828311.263249:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:0.0:1787828311.263323:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828311.263344:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828311.263388:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828311.263402:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828311.263420:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:1.0:1787828311.263428:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828308/real 1787828311] req@ffff976889cfa940 x1874670564256768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787828311.263441:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:1.0:1787828311.263475:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564256768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828311.263491:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828311.263499:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828311.263505:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564256768:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828311.263515:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828308/real 1787828311] req@ffff976889cfbc00 x1874670564256896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828311.263533:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564256896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828311.263544:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828311.263548:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828311.263552:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564256896:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828311.263561:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828308/real 1787828311] req@ffff976889cfa580 x1874670564256640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828311.263578:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564256640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828311.263588:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828311.263593:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828311.263598:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564256640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828311.263662:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828308/real 1787828311] req@ffff976889cf8000 x1874670564257024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828311.263679:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564257024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828311.263689:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828311.263693:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828311.263697:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564257024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828311.263705:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828308/real 1787828311] req@ffff976889cfb0c0 x1874670564257152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828311.263719:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564257152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828311.263729:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828311.263733:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828311.263737:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564257152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828311.518910:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828311.518910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828311.518936:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828312.287034:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828312.542893:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828312.542900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828312.543001:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828313.310930:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828313.310940:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828313.310944:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828313.310953:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3253 00000100:00100000:1.0:1787828313.310957:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828313.310963:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828313.310970:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828313.311001:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564257280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828313.311023:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828313.311026:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828313.311027:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828313.311030:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3253 00000100:00080000:0.0:1787828313.311032:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828313.311035:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828313.311044:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828313.311051:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564257280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828313.311056:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564257408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828313.311079:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828313.311081:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828313.311083:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828313.311086:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3253 00000100:00080000:0.0:1787828313.311089:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828313.311091:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828313.311095:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828313.311106:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564257536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828313.311116:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828313.311121:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828313.311123:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828313.311127:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3253 00000100:00080000:0.0:1787828313.311132:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828313.311133:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828313.311140:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828313.311147:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828313.311161:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564257664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828313.311171:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828313/real 1787828313] req@ffff976882aeed00 x1874670564257280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828313.311172:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828313.311181:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828313.311186:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828313.311196:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3253 00000100:00080000:0.0:1787828313.311201:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828313.311205:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564257280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828313.311209:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828313.311219:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828313.311223:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828313.311231:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828313.311237:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564257792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828313.311241:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564257280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828313.311281:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564257408:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828313.311308:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828313.311322:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828313/real 1787828313] req@ffff976882aec3c0 x1874670564257408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828313.311336:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564257408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828313.311347:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828313.311350:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828313.311354:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564257408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828313.311360:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564257536:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828313.311381:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828313.311394:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828313/real 1787828313] req@ffff976882aee580 x1874670564257536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828313.311407:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564257536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828313.311422:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828313.311425:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828313.311428:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564257536:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828313.311435:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564257664:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828313.311456:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828313.311468:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828313/real 1787828313] req@ffff976882aed680 x1874670564257664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828313.311479:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564257664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828313.311488:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828313.311491:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828313.311494:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564257664:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828313.311500:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564257792:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828313.311527:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828313.311539:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828313/real 1787828313] req@ffff976882aee940 x1874670564257792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828313.311551:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564257792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828313.311558:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828313.311562:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828313.311566:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564257792:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828313.566934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828313.566956:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828313.567029:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828314.335114:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828314.590916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828314.590916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828314.590961:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828315.358894:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828315.614913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828315.614942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828315.614946:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828316.383028:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828316.639021:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828316.639033:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828316.639187:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828317.406947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828317.662926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828317.662948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828317.662948:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828318.430950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828318.430992:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828318.430998:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828318.431002:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828318.431012:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3258 00000100:00080000:3.0:1787828318.431022:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828318.431046:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828318.431115:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564257920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828318.431137:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828318.431139:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828318.431141:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828318.431159:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3258 00000100:00080000:3.0:1787828318.431164:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828318.431169:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828318.431181:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828318.431190:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564257920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828318.431202:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564258048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828318.431219:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828318.431224:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828318.431226:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828318.431232:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3258 00000100:00080000:3.0:1787828318.431236:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828318.431242:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828318.431249:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828318.431252:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828318.431285:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564258176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828318.431300:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828318.431302:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828318.431304:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828318.431309:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3258 00000100:00100000:1.0:1787828318.431310:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828318/real 1787828318] req@ffff97688cb5c3c0 x1874670564257920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828318.431312:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828318.431319:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828318.431327:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828318.431333:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564257920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828318.431339:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564258304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828318.431345:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828318.431350:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828318.431354:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:3.0:1787828318.431355:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828318.431358:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828318.431359:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564257920:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787828318.431363:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3258 00000100:00080000:3.0:1787828318.431372:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828318.431375:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828318.431378:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828318.431387:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564258432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828318.431389:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564258048:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828318.431422:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828318.431437:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828318/real 1787828318] req@ffff97688cb5cf00 x1874670564258048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828318.431448:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564258048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828318.431455:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828318.431457:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828318.431459:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564258048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828318.431464:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564258176:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828318.431478:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828318.431485:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828318/real 1787828318] req@ffff97688cb5c780 x1874670564258176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828318.431493:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564258176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828318.431498:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828318.431500:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828318.431502:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564258176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828318.431505:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564258304:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828318.431516:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828318.431522:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828318/real 1787828318] req@ffff97688cb5e1c0 x1874670564258304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828318.431529:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564258304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828318.431534:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828318.431536:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828318.431538:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564258304:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828318.431555:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564258432:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828318.431573:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828318.431585:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828318/real 1787828318] req@ffff97688cb5d2c0 x1874670564258432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828318.431598:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564258432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828318.431606:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828318.431608:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828318.431610:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564258432:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828318.686948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828318.686991:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828318.687092:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828319.454920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828319.710938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828319.710938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828319.710969:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828320.478930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828320.734930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828320.734931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828320.735028:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828321.502906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828321.758911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828321.758918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828321.758954:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828322.526993:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828322.782914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828322.782915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828322.782969:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828323.550896:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828323.550950:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828323.550961:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828323.550969:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828323.550983:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3263 00000100:00080000:3.0:1787828323.551033:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828323.551056:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828323.551137:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564258560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828323.551176:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828323.551180:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828323.551183:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828323.551188:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3263 00000100:00080000:3.0:1787828323.551193:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828323.551209:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828323.551223:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828323.551240:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564258688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828323.551249:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564258560:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787828323.551262:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828323.551268:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828323.551271:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828323.551276:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3263 00000100:00080000:3.0:1787828323.551281:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828323.551285:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828323.551290:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828323.551316:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564258816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828323.551337:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828323.551341:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828323.551343:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828323.551349:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3263 00000800:00000100:1.0:1787828323.551358:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787828323.551361:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828323.551371:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828323.551378:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828323.551394:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564258944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828323.551405:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828323.551409:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828323.551411:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828323.551417:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3263 00000100:00080000:3.0:1787828323.551422:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828323.551428:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828323/real 1787828323] req@ffff97688cb5f0c0 x1874670564258560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828323.551432:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828323.551443:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828323.551467:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564258560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828323.551474:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564259072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828323.551526:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828323.551533:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828323.551540:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564258560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828323.551581:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564258688:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828323.551622:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828323.551646:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828323/real 1787828323] req@ffff97688cb5d680 x1874670564258688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828323.551669:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564258688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828323.551679:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828323.551683:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828323.551687:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564258688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828323.551694:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564258816:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828323.551720:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828323.551732:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828323/real 1787828323] req@ffff97688cb5f840 x1874670564258816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828323.551744:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564258816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828323.551753:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828323.551757:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828323.551760:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564258816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828323.551767:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564258944:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828323.551785:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828323.551797:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828323/real 1787828323] req@ffff97688cb5de00 x1874670564258944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828323.551818:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564258944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828323.551880:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828323.551887:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828323.551891:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564258944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828323.551898:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564259072:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828323.551926:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828323.551939:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828323/real 1787828323] req@ffff97688cb5cb40 x1874670564259072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828323.551952:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564259072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828323.551960:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828323.551964:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828323.551967:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564259072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828323.806970:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828323.806977:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828323.807039:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828324.574973:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828324.831006:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828324.831013:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828324.831150:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828325.598894:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828325.854926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828325.854949:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828325.854956:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828326.623004:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828326.878906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828326.878997:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828326.879077:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828327.646911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828327.902940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828327.902940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828327.902989:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828328.671021:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828328.671028:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828328.671040:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828328.671046:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828328.671057:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3268 00000100:00080000:2.0:1787828328.671070:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828328.671078:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828328.671130:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564259200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828328.671155:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828328.671158:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828328.671159:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828328.671164:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3268 00000100:00080000:2.0:1787828328.671168:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828328.671172:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828328.671182:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828328.671201:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564259328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828328.671214:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828328.671217:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828328.671218:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828328.671221:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3268 00000100:00080000:2.0:1787828328.671224:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828328.671229:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828328.671239:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564259200:192.168.203.139@tcp:250:kworker.0 00000008:00100000:2.0:1787828328.671239:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828328.671265:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564259456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828328.671279:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828328.671282:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828328.671283:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828328.671286:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3268 00000100:00080000:2.0:1787828328.671289:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828328.671292:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828328.671296:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828328.671308:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564259584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828328.671314:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828328.671317:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828328.671318:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828328.671321:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3268 00000100:00080000:2.0:1787828328.671324:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828328.671327:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828328.671330:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828328.671341:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564259712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828328.671364:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828328.671431:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828328/real 1787828328] req@ffff976889cf8f00 x1874670564259200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828328.671460:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564259200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828328.671478:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828328.671484:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828328.671491:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564259200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828328.671503:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564259328:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828328.671538:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828328.671557:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828328/real 1787828328] req@ffff976889cfb0c0 x1874670564259328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828328.671589:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564259328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828328.671602:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828328.671607:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828328.671612:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564259328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828328.671649:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564259456:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828328.671706:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828328.671730:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828328/real 1787828328] req@ffff976889cf8000 x1874670564259456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828328.671754:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564259456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828328.671769:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828328.671775:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828328.671781:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564259456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828328.671807:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564259584:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828328.671833:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828328.671864:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828328/real 1787828328] req@ffff976889cfa580 x1874670564259584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828328.672004:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564259584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828328.672018:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828328.672023:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828328.672028:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564259584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828328.672039:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564259712:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828328.672072:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828328.672091:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828328/real 1787828328] req@ffff976889cfbc00 x1874670564259712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828328.672109:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564259712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828328.672131:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828328.672137:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828328.672142:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564259712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828328.926910:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828328.926910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828328.926984:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828329.695018:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828329.950913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828329.950918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828329.950998:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828330.719004:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828330.975041:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828330.975050:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828330.975227:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828331.743033:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828331.998931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828331.998985:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828331.999029:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828332.767020:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828333.022904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828333.022917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828333.023090:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828333.790908:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828333.790962:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828333.790969:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828333.790974:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828333.790984:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3273 00000100:00080000:0.0:1787828333.790997:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828333.791004:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828333.791049:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564259840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828333.791076:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828333.791080:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828333.791082:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828333.791085:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3273 00000100:00080000:0.0:1787828333.791088:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828333.791092:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828333.791118:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828333.791127:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564259840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828333.791133:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564259968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828333.791142:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828333.791144:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828333.791146:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828333.791148:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3273 00000100:00080000:0.0:1787828333.791151:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828333.791153:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828333.791157:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828333.791182:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564260096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828333.791190:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828333.791193:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828333.791195:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828333.791198:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3273 00000100:00080000:0.0:1787828333.791209:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828333.791212:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828333.791216:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1787828333.791222:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828333.791227:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564260224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828333.791236:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828333.791255:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828333.791257:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828333.791261:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3273 00000100:00080000:0.0:1787828333.791264:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828333.791266:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828333.791269:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828333.791278:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564260352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828333.791286:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828333/real 1787828333] req@ffff976882aef0c0 x1874670564259840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828333.791319:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564259840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828333.791329:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828333.791333:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828333.791336:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564259840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828333.791361:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564259968:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828333.791380:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828333.791391:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828333/real 1787828333] req@ffff976882aee940 x1874670564259968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828333.791401:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564259968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828333.791408:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828333.791410:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828333.791413:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564259968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828333.791417:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564260096:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828333.791428:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828333.791435:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828333/real 1787828333] req@ffff976882aed680 x1874670564260096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828333.791453:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564260096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828333.791459:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828333.791462:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828333.791464:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564260096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828333.791468:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564260224:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828333.791481:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828333.791488:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828333/real 1787828333] req@ffff976882aee580 x1874670564260224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828333.791496:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564260224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828333.791502:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828333.791504:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828333.791506:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564260224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828333.791510:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564260352:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828333.791521:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828333.791527:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828333/real 1787828333] req@ffff976882aec3c0 x1874670564260352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828333.791534:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564260352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828333.791539:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828333.791541:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828333.791544:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564260352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828334.046972:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828334.046973:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828334.046993:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828334.814969:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828335.070992:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828335.070993:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828335.071098:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828335.838919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828336.095009:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828336.095040:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828336.095066:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828336.863052:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828337.118962:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828337.118985:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828337.119104:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828337.886982:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828338.142951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828338.142986:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828338.142992:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828338.911034:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828338.911056:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828338.911069:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828338.911077:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828338.911093:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3278 00000100:00080000:1.0:1787828338.911114:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828338.911130:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828338.911197:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564260480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828338.911239:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828338.911244:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828338.911247:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828338.911253:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3278 00000100:00080000:1.0:1787828338.911258:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828338.911265:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828338.911279:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564260480:192.168.203.139@tcp:250:kworker.0 00000008:00100000:1.0:1787828338.911280:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828338.911306:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564260608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828338.911321:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828338.911343:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828338.911347:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828338.911357:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3278 00000800:00000100:0.0:1787828338.911364:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787828338.911369:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828338.911382:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828338.911409:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828338.911418:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828338/real 1787828338] req@ffff976889efcb40 x1874670564260480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828338.911434:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564260736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828338.911437:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564260480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828338.911478:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828338.911493:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828338.911509:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564260480:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787828338.911531:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828338.911536:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828338.911540:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828338.911547:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564260608:192.168.203.139@tcp:38:kworker.0 00000100:00080000:1.0:1787828338.911549:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3278 00000100:00080000:1.0:1787828338.911562:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828338.911572:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:0.0:1787828338.911583:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787828338.911584:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828338.911600:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828338/real 1787828338] req@ffff976889efe1c0 x1874670564260608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828338.911615:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564260864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828338.911624:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564260608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828338.911691:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828338.911692:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828338.911719:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00080000:1.0:1787828338.911727:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787828338.911732:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564260608:192.168.203.139@tcp:38:kworker.0 00000100:00080000:1.0:1787828338.911734:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828338.911759:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3278 00000100:00100000:0.0:1787828338.911760:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564260736:192.168.203.139@tcp:38:kworker.0 00000100:00080000:1.0:1787828338.911790:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828338.911800:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:0.0:1787828338.911806:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787828338.911813:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828338.911839:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564260992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828338.911849:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828338/real 1787828338] req@ffff976889efe580 x1874670564260736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828338.911869:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564260736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828338.911878:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828338.911898:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828338.911901:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564260736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828338.911928:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564260864:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828338.911953:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828338.911962:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828338/real 1787828338] req@ffff976889eff480 x1874670564260864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828338.911969:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564260864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828338.911991:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828338.911994:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828338.911996:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564260864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828338.912000:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564260992:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828338.912010:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828338.912016:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828338/real 1787828338] req@ffff976889efc780 x1874670564260992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828338.912023:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564260992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828338.912028:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828338.912030:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828338.912032:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564260992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828339.166949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828339.167023:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828339.167167:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828339.934991:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828340.190982:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828340.191026:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828340.191036:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828340.959005:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828341.214938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828341.214946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828341.215011:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828341.982961:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828342.238958:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828342.238958:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828342.239012:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828343.006884:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828343.262958:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828343.262958:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828343.263117:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828344.030951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828344.030984:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828344.030993:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828344.030999:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828344.031010:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3283 00000100:00080000:2.0:1787828344.031023:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828344.031032:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828344.031074:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564261120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828344.031134:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828344.031138:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828344.031140:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828344.031145:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3283 00000100:00080000:2.0:1787828344.031150:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828344.031155:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828344.031167:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828344.031179:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564261120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828344.031189:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564261248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828344.031204:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828344.031208:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828344.031210:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828344.031214:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3283 00000100:00080000:2.0:1787828344.031218:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828344.031223:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828344.031229:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828344.031242:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564261376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828344.031251:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828344.031255:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828344.031260:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828344.031261:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828344.031268:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3283 00000100:00080000:2.0:1787828344.031275:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828344.031282:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828344.031291:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828344.031327:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564261504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828344.031344:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828344.031346:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828344/real 1787828344] req@ffff976889cfa940 x1874670564261120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828344.031357:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828344.031364:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828344.031376:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3283 00000100:00100000:1.0:1787828344.031382:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564261120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828344.031386:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828344.031415:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:1.0:1787828344.031420:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787828344.031422:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828344.031430:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828344.031438:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564261120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828344.031444:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564261632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828344.031484:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564261248:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828344.031525:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828344.031542:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828344/real 1787828344] req@ffff976889cfad00 x1874670564261248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828344.031557:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564261248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828344.031569:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828344.031574:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828344.031580:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564261248:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828344.031591:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564261376:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828344.031625:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828344.031650:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828344/real 1787828344] req@ffff976889cf92c0 x1874670564261376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828344.031674:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564261376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828344.031688:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828344.031692:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828344.031696:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564261376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828344.031703:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564261504:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828344.031733:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828344.031754:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828344/real 1787828344] req@ffff976889cf9e00 x1874670564261504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828344.031768:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564261504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828344.031777:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828344.031781:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828344.031787:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564261504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828344.031795:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564261632:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828344.031812:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828344.031821:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828344/real 1787828344] req@ffff976889cfb480 x1874670564261632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828344.031831:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564261632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828344.031903:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828344.031908:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828344.031913:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564261632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828344.286980:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828344.287008:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828344.287011:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828345.054902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828345.310947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828345.310950:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828345.311014:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828346.079012:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828346.335009:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828346.335031:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828346.335116:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828347.102997:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828347.358903:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828347.358927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828347.358975:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828348.126975:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828348.382984:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828348.383056:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828348.383093:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828349.151013:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828349.151055:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828349.151069:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828349.151076:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828349.151092:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3289 00000100:00080000:1.0:1787828349.151109:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828349.151121:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828349.151177:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564261760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828349.151224:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828349.151229:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828349.151231:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828349.151237:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3289 00000100:00080000:1.0:1787828349.151243:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828349.151249:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828349.151262:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828349.151279:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564261888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828349.151296:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828349.151301:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828349.151303:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828349.151311:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3289 00000100:00100000:0.0:1787828349.151317:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564261760:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787828349.151317:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828349.151333:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828349.151345:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828349.151359:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564262016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828349.151371:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828349.151375:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828349.151377:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828349.151383:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3289 00000100:00080000:1.0:1787828349.151386:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828349.151389:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828349.151396:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828349.151409:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564262144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828349.151420:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828349.151423:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828349.151426:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828349.151430:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3289 00000100:00080000:1.0:1787828349.151455:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828349.151463:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828349.151474:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828349.151476:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828349.151506:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564262272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828349.151567:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828349/real 1787828349] req@ffff976889efc780 x1874670564261760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828349.151597:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564261760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828349.151618:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828349.151623:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828349.151627:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564261760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828349.151658:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564261888:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828349.151684:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828349.151697:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828349/real 1787828349] req@ffff976889eff480 x1874670564261888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828349.151725:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564261888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828349.151737:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828349.151742:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828349.151746:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564261888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828349.151755:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564262016:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828349.151781:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828349.151793:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828349/real 1787828349] req@ffff976889efe580 x1874670564262016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828349.151906:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564262016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828349.151915:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828349.151918:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828349.151920:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564262016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828349.151926:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564262144:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828349.151948:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828349.151958:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828349/real 1787828349] req@ffff976889efe1c0 x1874670564262144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828349.151967:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564262144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828349.151975:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828349.151978:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828349.151980:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564262144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828349.151985:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564262272:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828349.151998:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828349.152004:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828349/real 1787828349] req@ffff976889efcb40 x1874670564262272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828349.152013:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564262272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828349.152025:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828349.152028:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828349.152030:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564262272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828349.406970:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828349.407037:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828349.407074:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828350.174937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828350.431008:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828350.431084:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828350.431094:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828351.198963:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828351.454945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828351.454947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828351.455100:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828352.222895:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828352.478918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828352.478919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828352.478954:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828353.246967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828353.502991:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828353.502991:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828353.503170:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828354.270986:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828354.271062:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828354.271072:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828354.271077:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828354.271090:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3294 00000100:00080000:0.0:1787828354.271103:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828354.271112:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828354.271158:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564262400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828354.271202:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828354.271207:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828354.271210:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828354.271216:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3294 00000100:00080000:0.0:1787828354.271221:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828354.271225:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828354.271239:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828354.271255:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564262528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828354.271271:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828354.271276:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828354.271281:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828354.271288:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3294 00000100:00080000:0.0:1787828354.271294:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828354.271306:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828354.271306:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564262400:192.168.203.139@tcp:250:kworker.0 00000008:00100000:0.0:1787828354.271328:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828354.271346:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564262656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828354.271361:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828354.271367:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828354.271371:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828354.271376:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3294 00000100:00080000:0.0:1787828354.271382:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828354.271387:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828354.271393:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828354.271405:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564262784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828354.271416:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828354.271420:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828354.271422:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828354.271428:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3294 00000100:00080000:0.0:1787828354.271435:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828354.271440:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828354.271448:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828354.271462:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564262912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828354.271493:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828354.271573:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828354/real 1787828354] req@ffff976882aeed00 x1874670564262400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828354.271615:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564262400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828354.271627:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828354.271631:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828354.271634:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564262400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828354.271642:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564262528:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828354.271662:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828354.271672:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828354/real 1787828354] req@ffff976882aec3c0 x1874670564262528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828354.271710:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564262528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828354.271720:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828354.271724:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828354.271728:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564262528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828354.271767:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564262656:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828354.271854:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828354.271863:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828354/real 1787828354] req@ffff976882aee580 x1874670564262656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828354.271872:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564262656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828354.271879:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828354.271882:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828354.271884:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564262656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828354.271889:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564262784:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828354.271903:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828354.271909:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828354/real 1787828354] req@ffff976882aed680 x1874670564262784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828354.271916:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564262784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828354.271927:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828354.271929:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828354.271931:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564262784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828354.271935:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564262912:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828354.271946:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828354.271952:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828354/real 1787828354] req@ffff976882aee940 x1874670564262912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828354.271958:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564262912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828354.271964:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828354.271966:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828354.271969:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564262912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828354.526979:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828354.526989:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828354.527007:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828355.294956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828355.550996:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828355.551026:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828355.551118:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828356.318911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828356.574979:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828356.574982:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828356.575094:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828357.342926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828357.598892:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828357.598916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828357.598989:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828358.366904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828358.622918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828358.622966:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828358.623015:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828359.390932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828359.391039:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828359.391050:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828359.391055:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828359.391069:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3299 00000100:00080000:1.0:1787828359.391097:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828359.391105:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828359.391149:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564263040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828359.391198:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828359.391202:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828359.391204:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828359.391211:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3299 00000100:00080000:1.0:1787828359.391217:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828359.391220:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828359.391230:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828359.391242:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564263040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828359.391247:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564263168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828359.391267:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828359.391271:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828359.391273:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828359.391278:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3299 00000100:00080000:1.0:1787828359.391286:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828359.391289:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828359.391294:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828359.391305:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564263296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828359.391312:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828359.391316:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828359.391318:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828359.391324:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3299 00000100:00080000:1.0:1787828359.391340:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828359.391345:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828359.391350:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1787828359.391357:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828359.391369:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564263424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828359.391381:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828359.391384:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828359.391387:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828359.391391:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3299 00000100:00080000:1.0:1787828359.391395:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828359.391398:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828359.391402:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828359.391414:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564263552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828359.391423:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828359/real 1787828359] req@ffff976889efe940 x1874670564263040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828359.391456:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564263040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828359.391468:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828359.391472:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828359.391476:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564263040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828359.391504:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564263168:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828359.391525:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828359.391536:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828359/real 1787828359] req@ffff976889eff0c0 x1874670564263168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828359.391553:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564263168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828359.391560:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828359.391562:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828359.391565:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564263168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828359.391570:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564263296:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828359.391586:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828359.391593:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828359/real 1787828359] req@ffff976889effc00 x1874670564263296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828359.391603:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564263296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828359.391609:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828359.391611:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828359.391613:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564263296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828359.391618:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564263424:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828359.391632:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828359.391640:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828359/real 1787828359] req@ffff976889efed00 x1874670564263424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828359.391649:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564263424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828359.391662:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828359.391665:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828359.391667:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564263424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828359.391672:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564263552:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828359.391684:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828359.391691:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828359/real 1787828359] req@ffff976889efc3c0 x1874670564263552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828359.391700:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564263552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828359.391707:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828359.391709:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828359.391712:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564263552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828359.646926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828359.646933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828359.646981:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828360.414953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828360.670949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828360.670974:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828360.671004:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828361.438989:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828361.694925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828361.694990:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828361.695040:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828362.462941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828362.718945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828362.718971:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828362.718994:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828363.486977:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828363.742923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828363.742968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828363.743067:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828364.510992:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828364.511005:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828364.511007:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828364.511015:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828364.511033:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3304 00000100:00080000:1.0:1787828364.511050:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828364.511063:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828364.511126:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564263680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828364.511170:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828364.511175:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828364.511178:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828364.511184:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3304 00000100:00080000:1.0:1787828364.511188:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828364.511193:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828364.511207:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828364.511222:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564263808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828364.511237:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828364.511242:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828364.511244:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828364.511250:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3304 00000100:00080000:1.0:1787828364.511254:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828364.511258:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828364.511265:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828364.511281:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564263936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828364.511288:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564263680:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787828364.511299:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828364.511308:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828364.511311:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828364.511317:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3304 00000100:00080000:1.0:1787828364.511321:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828364.511325:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828364.511332:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828364.511347:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564264064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828364.511359:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828364.511363:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828364.511365:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828364.511371:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3304 00000100:00080000:1.0:1787828364.511394:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828364.511399:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828364.511407:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828364.511417:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828364.511429:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564264192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828364.511498:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828364/real 1787828364] req@ffff976889efc3c0 x1874670564263680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828364.511533:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564263680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828364.511554:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828364.511561:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828364.511569:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564263680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828364.511583:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564263808:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828364.511629:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828364.511654:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828364/real 1787828364] req@ffff976889efed00 x1874670564263808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828364.511676:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564263808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828364.511693:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828364.511699:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828364.511705:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564263808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828364.511779:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564263936:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828364.511899:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828364.511924:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828364/real 1787828364] req@ffff976889effc00 x1874670564263936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828364.511968:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564263936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828364.511985:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828364.511992:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828364.511999:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564263936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828364.512012:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564264064:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828364.512051:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828364.512075:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828364/real 1787828364] req@ffff976889eff0c0 x1874670564264064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828364.512100:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564264064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828364.512117:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828364.512124:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828364.512131:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564264064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828364.512144:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564264192:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828364.512186:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828364.512209:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828364/real 1787828364] req@ffff976889efe940 x1874670564264192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828364.512246:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564264192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828364.512263:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828364.512270:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828364.512277:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564264192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828364.766969:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828364.766992:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828364.767055:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828365.534998:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828365.790990:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828365.790996:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828365.791031:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828366.558987:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828366.814934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828366.814946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828366.814996:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828367.582902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828367.838949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828367.838976:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828367.839031:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828368.606940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828368.862932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828368.862967:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828368.862988:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828369.630923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828369.630945:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828369.630952:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828369.630957:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828369.630969:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3309 00000100:00080000:0.0:1787828369.630979:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828369.630986:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828369.631025:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564264320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828369.631057:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828369.631061:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828369.631063:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828369.631066:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3309 00000100:00080000:0.0:1787828369.631075:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828369.631081:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828369.631094:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828369.631115:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564264448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828369.631123:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564264320:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828369.631146:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828369.631151:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828369.631154:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828369.631159:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3309 00000100:00080000:0.0:1787828369.631163:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828369.631167:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828369.631174:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828369.631188:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564264576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828369.631198:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828369.631202:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828369.631203:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828369.631207:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3309 00000100:00080000:0.0:1787828369.631211:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828369.631219:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:1.0:1787828369.631219:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:0.0:1787828369.631229:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828369.631240:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564264704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828369.631254:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828369.631258:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828369.631261:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828369.631266:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3309 00000100:00080000:0.0:1787828369.631272:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828369.631278:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828369.631286:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828369.631288:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828369/real 1787828369] req@ffff976882aef0c0 x1874670564264320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828369.631309:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564264832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828369.631353:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564264320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828369.631371:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828369.631379:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828369.631385:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564264320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828369.631417:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564264448:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828369.631452:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828369.631472:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828369/real 1787828369] req@ffff976882aee940 x1874670564264448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828369.631491:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564264448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828369.631505:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828369.631510:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828369.631515:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564264448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828369.631524:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564264576:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828369.631552:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828369.631570:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828369/real 1787828369] req@ffff976882aed680 x1874670564264576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828369.631605:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564264576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828369.631636:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828369.631643:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828369.631650:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564264576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828369.631706:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564264704:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828369.631758:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828369.631776:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828369/real 1787828369] req@ffff976882aee580 x1874670564264704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828369.631794:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564264704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828369.631807:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828369.631812:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828369.631817:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564264704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828369.631826:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564264832:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828369.631911:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828369.631931:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828369/real 1787828369] req@ffff976882aec3c0 x1874670564264832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828369.631951:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564264832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828369.631966:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828369.631971:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828369.631977:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564264832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828369.886975:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828369.886985:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828369.887018:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828370.654987:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828370.910930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828370.910960:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828370.910964:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828371.678960:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828371.934972:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828371.935018:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828371.935035:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828372.702907:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828372.959009:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828372.959012:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828372.959055:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828373.726909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828373.982911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828373.982916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828373.982944:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828374.750927:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828374.750930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828374.750941:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828374.750945:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828374.750955:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3314 00000100:00080000:0.0:1787828374.750965:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828374.750971:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828374.751010:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564264960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828374.751044:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828374.751048:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828374.751051:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828374.751056:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3314 00000100:00080000:0.0:1787828374.751060:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828374.751063:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828374.751073:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828374.751088:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564265088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828374.751100:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828374.751105:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828374.751107:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828374.751113:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3314 00000100:00080000:0.0:1787828374.751117:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828374.751123:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828374.751130:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828374.751135:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564264960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828374.751148:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564265216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828374.751164:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828374.751168:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828374.751170:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828374.751176:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3314 00000100:00080000:0.0:1787828374.751181:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828374.751186:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828374.751193:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828374.751208:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564265344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828374.751216:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828374.751219:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828374.751221:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828374.751224:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3314 00000100:00080000:0.0:1787828374.751227:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828374.751231:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828374.751235:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828374.751244:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564265472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828374.751257:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564265088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828374.751325:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564265216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828374.751346:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564265344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828374.751362:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564265472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828375.006913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828375.006920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828375.006960:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828375.774910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828376.030930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828376.030937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828376.030958:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828376.798933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828377.054963:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828377.054964:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828377.055025:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828377.822921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:2.0:1787828377.823122:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:2.0:1787828377.823160:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787828377.823173:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787828377.823208:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787828377.823221:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787828377.823230:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787828377.823241:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:1.0:1787828377.823257:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828374/real 1787828377] req@ffff976882aee580 x1874670564265216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828377.823287:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564265216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828377.823303:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828377.823309:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828377.823313:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564265216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828377.823323:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828374/real 1787828377] req@ffff976882aed680 x1874670564265344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828377.823339:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564265344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828377.823348:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828377.823351:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828377.823355:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564265344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828377.823363:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828374/real 1787828377] req@ffff976882aee940 x1874670564265472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828377.823377:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564265472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828377.823385:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828377.823389:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828377.823392:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564265472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828377.823399:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828374/real 1787828377] req@ffff976882aeed00 x1874670564264960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828377.823412:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564264960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828377.823420:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828377.823424:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828377.823427:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564264960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828377.823435:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828374/real 1787828377] req@ffff976882aec3c0 x1874670564265088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828377.823448:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564265088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828377.823457:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828377.823461:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828377.823464:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564265088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828378.078917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828378.078964:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828378.079001:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828378.846923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828379.102937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828379.102937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828379.102963:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828379.870949:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828379.870962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828379.870963:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828379.870971:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828379.870986:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3319 00000100:00080000:0.0:1787828379.870999:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828379.871012:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828379.871087:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564265600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828379.871152:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828379.871160:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828379.871165:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828379.871174:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3319 00000100:00080000:0.0:1787828379.871183:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828379.871192:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828379.871204:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564265600:192.168.203.139@tcp:250:kworker.0 00000008:00100000:0.0:1787828379.871210:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828379.871255:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564265728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828379.871279:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828379.871284:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828379.871288:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828379.871295:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3319 00000800:00000100:2.0:1787828379.871304:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828379.871309:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828379.871318:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828379.871326:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828379.871364:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564265856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828379.871386:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828379.871391:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828379/real 1787828379] req@ffff976882aef0c0 x1874670564265600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828379.871394:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828379.871403:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828379.871420:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3319 00000100:00080000:0.0:1787828379.871437:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828379.871447:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564265600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828379.871461:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:2.0:1787828379.871480:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828379.871482:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00080000:2.0:1787828379.871495:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828379.871505:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564265600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828379.871517:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564265984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828379.871547:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828379.871553:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828379.871558:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828379.871567:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564265728:192.168.203.139@tcp:38:kworker.0 00000100:00080000:0.0:1787828379.871574:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3319 00000100:00080000:0.0:1787828379.871588:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828379.871634:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828379.871647:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828379.871669:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828379.871677:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564266112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828379.871704:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828379/real 1787828379] req@ffff976882aec3c0 x1874670564265728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828379.871740:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564265728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828379.871767:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828379.871775:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828379.871781:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564265728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828379.871793:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564265856:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828379.871881:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828379.871906:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828379/real 1787828379] req@ffff976882aeed00 x1874670564265856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828379.871942:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564265856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828379.871959:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828379.871965:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828379.871971:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564265856:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828379.871982:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564265984:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828379.872022:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828379.872055:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828379/real 1787828379] req@ffff976882aee940 x1874670564265984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828379.872078:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564265984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828379.872092:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828379.872098:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828379.872106:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564265984:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828379.872138:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564266112:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828379.872177:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828379.872198:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828379/real 1787828379] req@ffff976882aed680 x1874670564266112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828379.872221:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564266112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828379.872237:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828379.872242:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828379.872249:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564266112:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828380.126914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828380.126915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828380.126933:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828380.894948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828381.150913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828381.150937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828381.150950:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828381.918986:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828382.174969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828382.174974:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828382.175033:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828382.942914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828383.198918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828383.198918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828383.198952:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828383.967056:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828384.222986:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828384.222987:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828384.223027:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828384.991003:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828384.991015:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828384.991017:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828384.991028:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828384.991041:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3324 00000100:00080000:3.0:1787828384.991055:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828384.991063:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828384.991120:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564266240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828384.991150:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828384.991154:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828384.991156:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828384.991161:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3324 00000100:00080000:3.0:1787828384.991164:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828384.991169:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828384.991196:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828384.991211:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564266368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828384.991223:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828384.991228:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828384.991230:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828384.991236:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564266240:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787828384.991239:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3324 00000100:00080000:3.0:1787828384.991249:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828384.991254:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828384.991259:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828384.991270:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564266496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828384.991279:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828384.991283:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828384.991285:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828384.991288:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3324 00000100:00080000:3.0:1787828384.991292:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828384.991295:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828384.991300:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828384.991314:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564266624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828384.991344:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828384.991353:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787828384.991356:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828384.991363:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828384.991371:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3324 00000100:00080000:3.0:1787828384.991379:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828384.991387:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828384.991395:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828384.991415:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828384/real 1787828384] req@ffff97688cb5c000 x1874670564266240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828384.991420:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564266752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828384.991491:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564266240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828384.991511:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828384.991516:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828384.991521:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564266240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828384.991551:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564266368:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828384.991605:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828384.991622:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828384/real 1787828384] req@ffff97688cb5cb40 x1874670564266368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828384.991636:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564266368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828384.991646:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828384.991649:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828384.991653:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564266368:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828384.991660:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564266496:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828384.991678:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828384.991688:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828384/real 1787828384] req@ffff97688cb5de00 x1874670564266496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828384.991699:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564266496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828384.991707:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828384.991710:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828384.991713:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564266496:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828384.991718:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564266624:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828384.991734:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828384.991743:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828384/real 1787828384] req@ffff97688cb5f840 x1874670564266624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828384.991754:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564266624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828384.991763:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828384.991766:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828384.991769:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564266624:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828384.991774:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564266752:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828384.991788:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828384.991797:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828384/real 1787828384] req@ffff97688cb5d680 x1874670564266752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828384.991807:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564266752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828384.991815:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828384.991817:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828384.991820:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564266752:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828385.246995:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828385.247019:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828385.247022:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828386.014894:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828386.270930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828386.270932:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828386.270937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828387.038967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828387.294985:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828387.294997:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828387.295059:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828388.062943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828388.318990:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828388.319006:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828388.319019:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828389.087039:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787828389.342968:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828389.342972:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828389.343026:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828390.110966:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828390.110966:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828390.110982:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828390.110989:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828390.111003:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3330 00000100:00080000:3.0:1787828390.111017:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828390.111028:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828390.111077:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564266880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828390.111134:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828390.111142:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828390.111146:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828390.111154:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3330 00000100:00080000:3.0:1787828390.111163:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828390.111172:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828390.111206:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828390.111230:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564266880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828390.111233:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564267008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828390.111292:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828390.111297:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828390.111301:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828390.111307:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3330 00000100:00080000:3.0:1787828390.111330:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828390.111336:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:2.0:1787828390.111344:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:3.0:1787828390.111346:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828390.111373:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564267136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828390.111406:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828390.111412:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828390.111419:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828390.111421:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828390/real 1787828390] req@ffff97688cb5f0c0 x1874670564266880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828390.111439:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3330 00000100:00080000:3.0:1787828390.111455:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828390.111464:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564266880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828390.111470:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828390.111492:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00080000:2.0:1787828390.111496:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828390.111506:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828390.111513:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564266880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828390.111525:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564267264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828390.111602:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564267008:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787828390.111611:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828390.111618:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828390.111621:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828390.111629:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3330 00000800:00000100:2.0:1787828390.111635:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787828390.111638:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828390.111648:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828390.111655:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828390/real 1787828390] req@ffff97688cb5d2c0 x1874670564267008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828390.111660:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828390.111676:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564267008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828390.111687:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828390.111690:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cf00 x1874670564267392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828390.111712:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828390.111727:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564267008:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828390.111748:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564267136:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828390.111810:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828390.111852:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828390/real 1787828390] req@ffff97688cb5e1c0 x1874670564267136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828390.111866:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564267136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828390.111875:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828390.111879:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828390.111882:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564267136:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828390.111901:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564267264:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828390.111934:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828390.111947:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828390/real 1787828390] req@ffff97688cb5c780 x1874670564267264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828390.111960:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564267264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828390.111968:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828390.111972:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828390.111975:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564267264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828390.111981:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564267392:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828390.111998:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828390.112007:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828390/real 1787828390] req@ffff97688cb5cf00 x1874670564267392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828390.112018:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cf00 x1874670564267392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828390.112026:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828390.112029:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828390.112031:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564267392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828390.366939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828390.366940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828390.367002:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828391.134924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828391.390904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828391.390918:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828391.390924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828392.158959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828392.414960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828392.414961:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828392.414994:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828393.182972:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828393.438944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828393.438951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828393.438995:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828394.206970:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828394.462948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828394.462972:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828394.463046:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828395.230901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828395.230952:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828395.230957:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828395.230960:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828395.230968:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3335 00000100:00080000:2.0:1787828395.230976:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828395.230982:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828395.231015:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564267520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828395.231061:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828395.231064:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828395.231065:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828395.231069:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3335 00000100:00080000:2.0:1787828395.231073:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828395.231077:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828395.231087:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828395.231099:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564267520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828395.231105:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564267648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828395.231132:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828395.231136:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828395.231138:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828395.231143:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3335 00000100:00080000:2.0:1787828395.231148:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828395.231153:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828395.231158:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828395.231172:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564267776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828395.231199:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828395.231209:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828395.231215:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828395.231218:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828395.231225:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3335 00000100:00080000:2.0:1787828395.231231:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828395.231235:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828395.231241:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828395.231245:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828395/real 1787828395] req@ffff976889cfa1c0 x1874670564267520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828395.231273:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564267904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828395.231283:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564267520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828395.231286:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828395.231303:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828395.231310:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828395.231313:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828395.231318:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3335 00000100:00080000:1.0:1787828395.231321:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:2.0:1787828395.231344:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828395.231345:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564267520:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828395.231355:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828395.231369:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828395.231383:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564268032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828395.231387:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564267648:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828395.231420:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828395.231434:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828395/real 1787828395] req@ffff976889cfb480 x1874670564267648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828395.231445:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564267648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828395.231452:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828395.231454:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828395.231457:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564267648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828395.231462:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564267776:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828395.231475:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828395.231483:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828395/real 1787828395] req@ffff976889cf9e00 x1874670564267776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828395.231492:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564267776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828395.231533:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828395.231535:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828395.231538:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564267776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828395.231542:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564267904:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828395.231558:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828395.231565:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828395/real 1787828395] req@ffff976889cf92c0 x1874670564267904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828395.231573:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564267904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828395.231578:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828395.231581:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828395.231584:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564267904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828395.231601:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564268032:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828395.231621:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828395.231634:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828395/real 1787828395] req@ffff976889cfad00 x1874670564268032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828395.231645:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564268032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828395.231650:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828395.231652:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828395.231655:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564268032:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828395.486931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828395.486936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828395.486991:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828396.254930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828396.510984:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828396.511000:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828396.511007:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828397.278986:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828397.534906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828397.534907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828397.534942:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828398.303007:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828398.558940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828398.558941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828398.558994:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828399.326901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828399.582917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828399.582917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828399.582962:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828400.350933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828400.350980:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828400.350986:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828400.350990:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828400.350999:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3340 00000100:00080000:0.0:1787828400.351008:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828400.351030:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828400.351070:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564268160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828400.351118:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828400.351120:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828400.351122:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828400.351125:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3340 00000100:00080000:0.0:1787828400.351128:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828400.351131:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828400.351144:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828400.351148:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564268160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828400.351159:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564268288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828400.351168:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828400.351170:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828400.351171:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828400.351174:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3340 00000100:00080000:0.0:1787828400.351176:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828400.351179:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828400.351182:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828400.351190:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564268416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828400.351197:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828400.351199:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828400.351200:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828400.351203:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3340 00000100:00080000:0.0:1787828400.351208:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828400.351208:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828400.351215:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828400.351219:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828400.351227:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564268544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828400.351234:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828400.351237:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828400.351238:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828400.351241:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3340 00000100:00080000:0.0:1787828400.351244:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828400.351250:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828400.351251:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828400/real 1787828400] req@ffff976882aee580 x1874670564268160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828400.351253:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828400.351265:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564268672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828400.351279:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564268160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828400.351293:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828400.351298:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828400.351303:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564268160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828400.351354:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564268288:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828400.351376:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828400.351386:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828400/real 1787828400] req@ffff976882aed680 x1874670564268288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828400.351396:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564268288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828400.351403:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828400.351405:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828400.351408:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564268288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828400.351412:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564268416:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828400.351423:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828400.351430:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828400/real 1787828400] req@ffff976882aee940 x1874670564268416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828400.351438:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564268416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828400.351444:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828400.351446:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828400.351448:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564268416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828400.351451:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564268544:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828400.351482:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828400.351518:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828400/real 1787828400] req@ffff976882aeed00 x1874670564268544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828400.351527:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564268544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828400.351533:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828400.351535:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828400.351537:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564268544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828400.351541:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564268672:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828400.351552:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828400.351558:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828400/real 1787828400] req@ffff976882aec3c0 x1874670564268672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828400.351565:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564268672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828400.351570:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828400.351572:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828400.351574:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564268672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828400.606930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828400.606930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828400.606960:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828401.375073:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828401.630994:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828401.631021:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828401.631034:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828402.398947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828402.654952:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828402.654952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828402.654996:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828403.422976:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828403.678925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828403.678936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828403.678995:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828404.446887:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828404.702920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828404.702961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828404.702967:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828405.470940:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828405.470940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828405.470951:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828405.470955:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828405.470967:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3345 00000100:00080000:2.0:1787828405.470980:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828405.470989:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828405.471033:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564268800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828405.471072:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828405.471078:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828405.471080:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828405.471084:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3345 00000100:00080000:2.0:1787828405.471088:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828405.471091:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828405.471101:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828405.471114:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564268928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828405.471125:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828405.471128:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828405.471130:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828405.471136:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3345 00000100:00100000:1.0:1787828405.471139:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564268800:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828405.471139:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828405.471146:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828405.471155:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828405.471183:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564269056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828405.471192:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828405.471195:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828405.471197:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828405.471201:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3345 00000100:00080000:2.0:1787828405.471204:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828405.471207:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828405.471213:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828405.471226:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564269184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828405.471241:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828405.471259:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828405.471266:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828405.471270:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828405.471277:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3345 00000100:00080000:2.0:1787828405.471284:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828405.471289:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828405.471297:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828405.471309:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828405/real 1787828405] req@ffff976889cfa940 x1874670564268800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828405.471317:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564269312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828405.471361:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564268800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828405.471374:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828405.471378:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828405.471382:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564268800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828405.471404:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564268928:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828405.471468:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828405.471482:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828405/real 1787828405] req@ffff976889cfbc00 x1874670564268928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828405.471494:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564268928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828405.471502:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828405.471505:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828405.471508:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564268928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828405.471513:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564269056:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828405.471530:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828405.471538:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828405/real 1787828405] req@ffff976889cfa580 x1874670564269056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828405.471548:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564269056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828405.471561:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828405.471564:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828405.471567:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564269056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828405.471571:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564269184:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828405.471595:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828405.471607:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828405/real 1787828405] req@ffff976889cf8000 x1874670564269184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828405.471623:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564269184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828405.471635:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828405.471640:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828405.471645:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564269184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828405.471653:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564269312:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828405.471674:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828405.471684:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828405/real 1787828405] req@ffff976889cfb0c0 x1874670564269312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828405.471703:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564269312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828405.471711:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828405.471713:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828405.471717:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564269312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828405.726982:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828405.727021:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828405.727033:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828406.494944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828406.750913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828406.750913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828406.750964:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828407.518908:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828407.774898:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828407.774906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828407.774951:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828408.542910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828408.799002:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828408.799002:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828408.799104:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828409.566955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828409.822946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828409.822951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828409.822985:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828410.590934:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828410.590943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828410.590945:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828410.590952:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828410.590962:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3350 00000100:00080000:2.0:1787828410.590992:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828410.591000:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828410.591037:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564269440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828410.591065:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828410.591068:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828410.591069:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828410.591073:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3350 00000100:00080000:2.0:1787828410.591076:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828410.591079:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828410.591088:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828410.591101:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564269568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828410.591128:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828410.591133:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828410.591134:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828410.591141:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3350 00000100:00100000:1.0:1787828410.591143:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564269440:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828410.591145:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828410.591155:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828410.591161:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828410.591176:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564269696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828410.591207:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828410.591212:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828410.591216:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828410.591222:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3350 00000100:00080000:2.0:1787828410.591229:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828410.591236:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828410.591244:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1787828410.591250:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828410.591269:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564269824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828410.591295:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828410.591298:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828410.591300:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828410.591305:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3350 00000100:00080000:2.0:1787828410.591310:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828410.591314:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828410.591321:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828410.591332:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828410/real 1787828410] req@ffff976889cf8f00 x1874670564269440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828410.591339:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564269952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828410.591371:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564269440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828410.591406:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828410.591412:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828410.591444:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564269440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828410.591454:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564269568:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828410.591479:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828410.591502:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828410/real 1787828410] req@ffff976889cfb0c0 x1874670564269568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828410.591513:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564269568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828410.591521:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828410.591523:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828410.591527:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564269568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828410.591551:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564269696:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828410.591565:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828410.591574:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828410/real 1787828410] req@ffff976889cf8000 x1874670564269696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828410.591584:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564269696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828410.591591:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828410.591594:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828410.591597:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564269696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828410.591601:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564269824:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828410.591613:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828410.591621:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828410/real 1787828410] req@ffff976889cfa580 x1874670564269824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828410.591629:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564269824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828410.591636:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828410.591638:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828410.591641:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564269824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828410.591645:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564269952:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828410.591657:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828410.591664:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828410/real 1787828410] req@ffff976889cfbc00 x1874670564269952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828410.591673:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564269952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828410.591679:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828410.591681:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828410.591683:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564269952:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828410.846942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828410.846943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828410.847017:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828411.614981:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828411.870983:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828411.871016:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828411.871052:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828412.638970:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828412.894972:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828412.894976:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828412.895115:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828413.662986:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828413.919004:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828413.919004:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828413.919208:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828414.687047:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828414.942955:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828414.943141:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828414.943184:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00080000:2.0:1787828415.710978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828415.710995:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828415.711002:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828415.711019:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3355 00000100:00080000:2.0:1787828415.711036:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828415.711049:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828415.711060:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828415.711096:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564270080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828415.711129:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828415.711134:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828415.711152:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828415.711163:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564270080:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828415.711164:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3355 00000100:00080000:2.0:1787828415.711180:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828415.711192:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828415.711209:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828415.711238:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564270208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828415.711256:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828415.711257:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828415.711265:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828415.711269:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828415.711281:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3355 00000100:00080000:2.0:1787828415.711294:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828415.711305:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828415.711316:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828415.711339:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564270336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828415.711345:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828415/real 1787828415] req@ffff976889cfa940 x1874670564270080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828415.711382:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828415.711393:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828415.711398:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828415.711407:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3355 00000100:00080000:2.0:1787828415.711413:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828415.711418:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828415.711426:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828415.711428:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564270080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828415.711450:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828415.711451:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564270464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828415.711464:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:2.0:1787828415.711478:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828415.711479:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564270080:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828415.711488:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828415.711493:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828415.711498:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3355 00000100:00080000:2.0:1787828415.711502:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828415.711506:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828415.711510:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828415.711519:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564270592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828415.711863:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564270208:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828415.711986:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828415.712014:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828415/real 1787828415] req@ffff976889cfad00 x1874670564270208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828415.712049:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564270208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828415.712064:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828415.712070:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828415.712076:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564270208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828415.712087:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564270336:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828415.712123:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828415.712142:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828415/real 1787828415] req@ffff976889cf92c0 x1874670564270336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828415.712163:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564270336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828415.712177:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828415.712182:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828415.712188:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564270336:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828415.712198:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564270464:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828415.712229:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828415.712246:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828415/real 1787828415] req@ffff976889cf9e00 x1874670564270464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828415.712265:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564270464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828415.712287:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828415.712292:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828415.712297:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564270464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828415.712306:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564270592:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828415.712335:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828415.712375:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828415/real 1787828415] req@ffff976889cfb480 x1874670564270592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828415.712415:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564270592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828415.712429:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828415.712434:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828415.712440:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564270592:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828415.966941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828415.966942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828415.967029:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828416.734970:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828416.990946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828416.990979:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828416.991024:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828417.758894:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828418.014970:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828418.014974:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828418.015088:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828418.782967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828419.038919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828419.038983:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828419.039100:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828419.806945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828420.063001:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828420.063002:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828420.063207:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828420.830931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828420.831029:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828420.831050:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828420.831058:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828420.831072:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3360 00000100:00080000:0.0:1787828420.831085:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828420.831095:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828420.831154:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564270720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828420.831193:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828420.831200:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828420.831204:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828420.831212:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3360 00000100:00080000:0.0:1787828420.831219:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828420.831226:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828420.831242:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828420.831258:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564270720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828420.831268:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564270848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828420.831288:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828420.831294:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828420.831297:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828420.831304:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3360 00000100:00080000:0.0:1787828420.831310:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828420.831336:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828420.831357:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828420.831378:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564270976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828420.831394:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828420.831400:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828420.831403:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787828420.831407:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828420.831411:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3360 00000100:00080000:0.0:1787828420.831416:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828420.831422:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828420.831431:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828420.831448:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564271104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828420.831465:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828420/real 1787828420] req@ffff976882aef0c0 x1874670564270720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828420.831466:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828420.831473:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828420.831476:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828420.831483:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3360 00000100:00080000:0.0:1787828420.831488:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828420.831495:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828420.831497:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564270720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828420.831503:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787828420.831517:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828420.831519:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564271232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828420.831526:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828420.831534:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564270720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828420.831606:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564270848:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828420.831659:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828420.831678:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828420/real 1787828420] req@ffff976882aec3c0 x1874670564270848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828420.831703:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564270848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828420.831714:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828420.831718:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828420.831723:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564270848:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828420.831731:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564270976:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828420.831752:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828420.831764:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828420/real 1787828420] req@ffff976882aeed00 x1874670564270976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828420.831778:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564270976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828420.831787:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828420.831791:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828420.831795:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564270976:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828420.831801:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564271104:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828420.831852:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828420.831868:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828420/real 1787828420] req@ffff976882aee940 x1874670564271104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828420.831883:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564271104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828420.831893:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828420.831897:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828420.831901:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564271104:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828420.831907:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564271232:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828420.831930:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828420.831941:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828420/real 1787828420] req@ffff976882aed680 x1874670564271232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828420.831969:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564271232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828420.831979:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828420.831983:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828420.831987:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564271232:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828421.086959:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828421.086961:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828421.087100:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828421.854939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828422.110974:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828422.111008:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828422.111041:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828422.878893:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828423.134888:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828423.134889:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828423.134985:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828423.902922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828424.158886:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828424.158915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828424.158988:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828424.926888:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828425.182889:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828425.182943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828425.182974:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828425.950961:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828425.951041:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828425.951049:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828425.951054:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828425.951067:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3365 00000100:00080000:1.0:1787828425.951081:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828425.951090:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828425.951141:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564271360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828425.951201:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828425.951206:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828425.951208:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828425.951213:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3365 00000100:00080000:1.0:1787828425.951218:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828425.951222:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828425.951235:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828425.951251:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564271488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828425.951267:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828425.951274:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787828425.951276:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564271360:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787828425.951278:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828425.951317:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3365 00000100:00080000:1.0:1787828425.951324:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828425.951329:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828425.951335:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828425.951357:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564271616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828425.951370:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828425.951375:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828425.951378:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828425.951383:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3365 00000100:00080000:1.0:1787828425.951399:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828425.951404:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828425.951411:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787828425.951422:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828425.951445:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564271744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828425.951461:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828425.951465:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828425.951467:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828425.951472:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3365 00000100:00080000:1.0:1787828425.951477:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828425.951482:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828425.951487:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828425.951500:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564271872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828425.951523:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828425/real 1787828425] req@ffff976889efcb40 x1874670564271360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828425.951580:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564271360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828425.951602:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828425.951626:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828425.951632:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564271360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828425.951669:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564271488:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828425.951710:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828425.951734:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828425/real 1787828425] req@ffff976889efe1c0 x1874670564271488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828425.951758:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564271488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828425.951773:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828425.951779:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828425.951784:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564271488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828425.951795:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564271616:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828425.951828:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828425.951893:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828425/real 1787828425] req@ffff976889efe580 x1874670564271616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828425.951933:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564271616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828425.951963:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828425.951970:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828425.951977:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564271616:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828425.951989:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564271744:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828425.952033:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828425.952058:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828425/real 1787828425] req@ffff976889eff480 x1874670564271744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828425.952084:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564271744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828425.952101:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828425.952109:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828425.952115:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564271744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828425.952128:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564271872:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828425.952167:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828425.952189:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828425/real 1787828425] req@ffff976889efc780 x1874670564271872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828425.952216:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564271872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828425.952233:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828425.952239:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828425.952247:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564271872:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828426.206943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828426.206977:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828426.207136:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828426.974972:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828427.230932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828427.230973:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828427.230995:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828427.998894:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828428.254893:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828428.254929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828428.255008:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828429.022989:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828429.278919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828429.278945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828429.279004:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828430.046944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828430.302937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828430.303042:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828430.303076:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828431.070922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828431.070930:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828431.070939:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828431.070944:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828431.070955:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3370 00000100:00080000:1.0:1787828431.070978:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828431.070986:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828431.071029:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564272000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828431.071076:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828431.071082:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828431.071086:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828431.071093:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3370 00000100:00080000:1.0:1787828431.071101:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828431.071109:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828431.071125:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828431.071128:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564272000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828431.071146:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564272128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828431.071168:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828431.071172:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828431.071175:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828431.071179:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3370 00000100:00080000:1.0:1787828431.071185:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828431.071190:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828431.071196:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828431.071204:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828431.071214:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564272256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828431.071232:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828431.071238:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828431.071241:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828431.071267:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3370 00000100:00100000:3.0:1787828431.071275:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828431/real 1787828431] req@ffff976889efc780 x1874670564272000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828431.071278:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828431.071294:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828431.071306:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828431.071308:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564272000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828431.071325:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828431.071330:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564272384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828431.071335:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828431.071347:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564272000:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787828431.071360:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828431.071364:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828431.071367:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828431.071374:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3370 00000100:00080000:1.0:1787828431.071380:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828431.071388:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828431.071388:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564272128:192.168.203.139@tcp:38:kworker.0 00000008:00100000:1.0:1787828431.071402:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828431.071418:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564272512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828431.071435:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828431.071454:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828431/real 1787828431] req@ffff976889eff480 x1874670564272128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828431.071476:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564272128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828431.071490:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828431.071496:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828431.071501:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564272128:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828431.071512:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564272256:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828431.071548:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828431.071564:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828431/real 1787828431] req@ffff976889efe580 x1874670564272256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828431.071577:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564272256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828431.071586:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828431.071600:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828431.071604:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564272256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828431.071611:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564272384:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828431.071632:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828431.071647:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828431/real 1787828431] req@ffff976889efe1c0 x1874670564272384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828431.071666:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564272384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828431.071680:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828431.071686:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828431.071691:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564272384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828431.071724:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564272512:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828431.071760:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828431.071772:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828431/real 1787828431] req@ffff976889efcb40 x1874670564272512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828431.071792:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564272512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828431.071801:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828431.071804:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828431.071807:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564272512:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828431.326964:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828431.326964:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828431.327074:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828432.094897:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828432.350969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828432.350977:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828432.351011:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828433.118915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828433.374958:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828433.374959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828433.375065:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828434.142980:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828434.398904:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828434.398935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828434.399048:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828435.166981:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828435.422922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828435.422954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828435.423063:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828436.191046:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828436.191060:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828436.191075:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828436.191084:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828436.191118:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3376 00000100:00080000:2.0:1787828436.191145:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828436.191156:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828436.191276:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564272640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828436.191307:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828436.191311:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828436.191313:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828436.191329:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3376 00000100:00080000:2.0:1787828436.191333:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828436.191338:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828436.191351:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828436.191358:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564272640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828436.191373:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564272768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828436.191391:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828436.191394:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828436.191396:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828436.191400:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3376 00000100:00080000:2.0:1787828436.191420:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828436.191424:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828436.191428:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828436.191439:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564272896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828436.191448:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828436.191451:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828436.191452:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828436.191456:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3376 00000100:00080000:2.0:1787828436.191460:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787828436.191463:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828436.191467:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828436.191474:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828436.191485:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564273024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828436.191494:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828436.191497:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828436.191499:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828436.191502:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3376 00000100:00080000:2.0:1787828436.191506:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828436.191510:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828436.191514:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828436.191522:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564273152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828436.191540:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828436/real 1787828436] req@ffff976889cfa1c0 x1874670564272640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828436.191562:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564272640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828436.191574:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828436.191579:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828436.191583:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564272640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828436.191625:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564272768:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828436.191649:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828436.191661:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828436/real 1787828436] req@ffff976889cfb480 x1874670564272768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828436.191674:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564272768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828436.191681:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828436.191684:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828436.191687:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564272768:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828436.191692:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564272896:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828436.191706:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828436.191714:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828436/real 1787828436] req@ffff976889cf9e00 x1874670564272896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828436.191724:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564272896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828436.191730:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828436.191733:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828436.191735:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564272896:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828436.191739:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564273024:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828436.191753:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828436.191761:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828436/real 1787828436] req@ffff976889cf92c0 x1874670564273024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828436.191771:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564273024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828436.191784:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828436.191787:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828436.191790:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564273024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828436.191794:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564273152:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828436.191806:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828436.191813:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828436/real 1787828436] req@ffff976889cfad00 x1874670564273152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828436.191845:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564273152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828436.191853:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828436.191856:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828436.191859:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564273152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828436.446975:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828436.447048:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828436.447095:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828437.215081:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828437.470999:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828437.471021:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828437.471171:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828438.238939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828438.494948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828438.494958:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828438.495013:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828439.262973:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828439.518967:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828439.518978:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828439.519102:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828440.287029:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828440.543024:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828440.543052:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828440.543112:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828441.310946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828441.311028:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828441.311039:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828441.311045:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828441.311056:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3381 00000100:00080000:0.0:1787828441.311069:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828441.311077:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828441.311142:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564273280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828441.311201:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828441.311208:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828441.311211:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828441.311220:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3381 00000100:00080000:0.0:1787828441.311228:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828441.311235:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828441.311249:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828441.311284:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564273408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828441.311285:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564273280:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828441.311328:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828441.311334:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828441.311338:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828441.311346:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3381 00000100:00080000:0.0:1787828441.311350:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828441.311356:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828441.311361:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828441.311376:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564273536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828441.311390:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828441.311394:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828441.311397:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828441.311402:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3381 00000100:00080000:0.0:1787828441.311420:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828441.311429:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828441.311439:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828441.311470:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564273664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828441.311493:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828441.311502:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828441.311507:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828441.311514:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564273408:192.168.203.139@tcp:38:kworker.0 00000100:00080000:0.0:1787828441.311529:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3381 00000100:00080000:0.0:1787828441.311539:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828441.311547:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828441.311561:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828441.311601:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564273792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828441.311621:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564273536:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828441.311701:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564273664:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828441.311750:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564273792:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828441.566997:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828441.567042:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828441.567097:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828442.334991:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828442.591030:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828442.591055:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828442.591128:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828443.358915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828443.614994:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828443.615072:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828443.615075:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828444.382921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:1.0:1787828444.383379:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:1.0:1787828444.383460:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787828444.383490:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787828444.383564:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787828444.383583:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787828444.383599:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787828444.383612:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:3.0:1787828444.383637:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828441/real 1787828444] req@ffff976882aed680 x1874670564273408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828444.383677:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564273408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828444.383692:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828444.383699:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828444.383704:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564273408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828444.383712:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828441/real 1787828444] req@ffff976882aee940 x1874670564273536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828444.383725:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564273536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828444.383733:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828444.383737:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828444.383740:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564273536:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828444.383748:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828441/real 1787828444] req@ffff976882aeed00 x1874670564273664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828444.383760:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564273664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828444.383768:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828444.383772:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828444.383775:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564273664:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828444.383781:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828441/real 1787828444] req@ffff976882aee580 x1874670564273280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828444.383795:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564273280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828444.383802:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828444.383806:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828444.383809:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564273280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828444.383888:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828441/real 1787828444] req@ffff976882aec3c0 x1874670564273792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828444.384087:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564273792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828444.384098:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828444.384107:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828444.384111:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564273792:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828444.638972:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828444.638978:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828444.639133:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828445.406975:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828445.662914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828445.662949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828445.662992:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828446.430895:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828446.430909:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828446.430915:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828446.430919:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828446.430930:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3386 00000100:00080000:0.0:1787828446.430941:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828446.430949:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828446.430988:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564273920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828446.431016:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828446.431020:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828446.431027:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828446.431032:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3386 00000100:00080000:0.0:1787828446.431037:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828446.431041:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828446.431053:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828446.431069:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564274048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828446.431073:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564273920:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828446.431080:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828446.431084:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828446.431085:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828446.431089:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3386 00000100:00080000:0.0:1787828446.431092:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828446.431096:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828446.431100:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828446.431110:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564274176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828446.431119:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828446.431122:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828446.431124:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828446.431127:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3386 00000100:00080000:0.0:1787828446.431130:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828446.431134:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828446.431152:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828446.431163:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564274304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828446.431168:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828446.431179:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828446.431183:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828446.431185:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828446.431188:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3386 00000100:00080000:0.0:1787828446.431191:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828446.431194:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828446.431199:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828446.431208:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564274432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828446.431216:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828446/real 1787828446] req@ffff976882aef0c0 x1874670564273920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828446.431238:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564273920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828446.431251:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828446.431256:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828446.431262:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564273920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828446.431297:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564274048:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828446.431327:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828446.431343:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828446/real 1787828446] req@ffff976882aec3c0 x1874670564274048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828446.431358:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564274048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828446.431367:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828446.431371:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828446.431374:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564274048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828446.431381:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564274176:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828446.431403:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828446.431415:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828446/real 1787828446] req@ffff976882aee580 x1874670564274176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828446.431439:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564274176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828446.431449:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828446.431453:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828446.431456:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564274176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828446.431462:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564274304:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828446.431485:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828446.431500:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828446/real 1787828446] req@ffff976882aeed00 x1874670564274304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828446.431514:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564274304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828446.431523:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828446.431525:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828446.431527:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564274304:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828446.431532:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564274432:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828446.431554:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828446.431566:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828446/real 1787828446] req@ffff976882aee940 x1874670564274432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828446.431579:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564274432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828446.431587:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828446.431591:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828446.431594:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564274432:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828446.686928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828446.686934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828446.686999:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828447.454902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828447.710913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828447.710924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828447.710983:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828448.478954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828448.734995:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828448.734995:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828448.735087:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828449.502958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828449.758926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828449.759304:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828449.759352:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828450.526972:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828450.782964:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828450.783011:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828450.783182:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828451.550999:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828451.551000:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828451.551016:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828451.551034:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828451.551047:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3391 00000100:00080000:0.0:1787828451.551061:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828451.551071:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828451.551175:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564274560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828451.551212:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828451.551216:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828451.551219:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828451.551225:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3391 00000100:00080000:0.0:1787828451.551229:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828451.551234:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828451.551246:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828451.551265:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564274688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828451.551279:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828451.551283:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828451.551286:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828451.551296:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564274560:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828451.551297:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3391 00000100:00080000:0.0:1787828451.551305:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828451.551315:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828451.551323:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828451.551336:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564274816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828451.551347:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828451.551351:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828451.551353:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828451.551380:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3391 00000100:00080000:0.0:1787828451.551385:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828451.551391:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828451.551408:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787828451.551427:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828451.551430:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564274944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828451.551447:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828451.551451:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828451.551454:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828451.551459:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3391 00000100:00080000:0.0:1787828451.551463:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828451.551469:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828451.551475:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828451.551508:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828451/real 1787828451] req@ffff976882aed680 x1874670564274560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828451.551509:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564275072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828451.551585:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564274560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828451.551604:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828451.551612:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828451.551620:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564274560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828451.551683:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564274688:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828451.551737:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828451.551778:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828451/real 1787828451] req@ffff976882aee940 x1874670564274688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828451.551799:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564274688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828451.551812:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828451.551817:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828451.551822:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564274688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828451.551832:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564274816:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828451.551983:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828451.552176:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828451/real 1787828451] req@ffff976882aeed00 x1874670564274816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828451.552418:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564274816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828451.552435:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828451.552440:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828451.552446:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564274816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828451.552459:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564274944:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828451.552577:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828451.552596:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828451/real 1787828451] req@ffff976882aee580 x1874670564274944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828451.552613:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564274944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828451.552629:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828451.552636:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828451.552643:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564274944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828451.552656:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564275072:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828451.552710:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828451.552731:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828451/real 1787828451] req@ffff976882aec3c0 x1874670564275072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828451.552750:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564275072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828451.552760:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828451.552764:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828451.552784:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564275072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828451.806963:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828451.806964:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828451.807171:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828452.574973:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828452.830958:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828452.830983:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828452.831013:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828453.598952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828453.854950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828453.854957:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828453.855111:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828454.622967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828454.879000:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828454.879026:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828454.879037:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828455.646894:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828455.902971:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828455.902985:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828455.903025:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828456.670927:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828456.670927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828456.670941:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828456.670946:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828456.670956:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3396 00000100:00080000:0.0:1787828456.670967:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828456.670974:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828456.671014:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564275200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828456.671043:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828456.671048:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828456.671051:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828456.671057:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3396 00000100:00080000:0.0:1787828456.671062:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828456.671084:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828456.671097:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828456.671118:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564275200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828456.671126:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564275328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828456.671139:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828456.671143:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828456.671145:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828456.671149:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3396 00000100:00080000:0.0:1787828456.671154:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828456.671158:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828456.671164:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828456.671185:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564275456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828456.671195:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828456.671199:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1787828456.671200:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828456.671201:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828456.671217:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3396 00000100:00080000:0.0:1787828456.671224:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828456.671228:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828456.671234:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828456.671248:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564275584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828456.671249:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828456/real 1787828456] req@ffff976882aef0c0 x1874670564275200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828456.671262:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828456.671266:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828456.671269:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828456.671273:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3396 00000100:00100000:3.0:1787828456.671273:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564275200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828456.671277:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828456.671291:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828456.671296:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787828456.671297:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828456.671319:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828456.671327:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564275200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828456.671342:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564275712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828456.671366:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564275328:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828456.671412:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828456.671438:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828456/real 1787828456] req@ffff976882aec3c0 x1874670564275328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828456.671455:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564275328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828456.671467:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828456.671473:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828456.671480:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564275328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828456.671493:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564275456:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828456.671534:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828456.671558:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828456/real 1787828456] req@ffff976882aee580 x1874670564275456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828456.671603:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564275456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828456.671614:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828456.671618:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828456.671622:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564275456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828456.671639:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564275584:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828456.671667:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828456.671682:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828456/real 1787828456] req@ffff976882aeed00 x1874670564275584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828456.671711:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564275584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828456.671722:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828456.671726:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828456.671730:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564275584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828456.671737:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564275712:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828456.671758:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828456.671770:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828456/real 1787828456] req@ffff976882aee940 x1874670564275712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828456.671784:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564275712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828456.671802:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828456.671805:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828456.671809:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564275712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828456.926979:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828456.926980:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828456.927021:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828457.694969:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828457.950983:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828457.951023:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828457.951093:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828458.719022:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828458.974973:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828458.974973:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828458.975033:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828459.743003:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828459.998919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828459.998964:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828459.999025:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828460.767018:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828461.022927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828461.022961:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828461.022966:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828461.790867:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828461.790878:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828461.790884:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828461.790888:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828461.790895:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3401 00000100:00080000:0.0:1787828461.790904:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828461.790909:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828461.790939:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564275840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828461.790965:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828461.790969:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828461.790971:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828461.790975:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3401 00000100:00080000:0.0:1787828461.790980:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828461.790985:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828461.790995:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828461.791011:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564275840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828461.791012:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564275968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828461.791041:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828461.791045:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828461.791047:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828461.791052:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3401 00000100:00080000:0.0:1787828461.791057:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828461.791061:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828461.791068:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828461.791082:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564276096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828461.791093:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828461.791094:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828461.791100:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828461.791103:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828461.791110:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3401 00000100:00080000:0.0:1787828461.791123:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828461.791128:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828461.791134:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828461.791137:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828461/real 1787828461] req@ffff976882aed680 x1874670564275840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828461.791151:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564276224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828461.791160:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564275840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828461.791188:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828461.791192:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828461.791194:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828461.791200:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828461.791202:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:0.0:1787828461.791209:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3401 00000100:00100000:3.0:1787828461.791211:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564275840:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828461.791214:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828461.791222:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828461.791227:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828461.791239:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564276352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828461.791242:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564275968:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828461.791279:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828461.791292:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828461/real 1787828461] req@ffff976882aee940 x1874670564275968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828461.791305:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564275968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828461.791319:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828461.791323:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828461.791327:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564275968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828461.791333:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564276096:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828461.791353:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828461.791364:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828461/real 1787828461] req@ffff976882aeed00 x1874670564276096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828461.791376:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564276096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828461.791384:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828461.791388:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828461.791391:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564276096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828461.791396:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564276224:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828461.791416:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828461.791427:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828461/real 1787828461] req@ffff976882aee580 x1874670564276224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828461.791438:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564276224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828461.791446:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828461.791450:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828461.791454:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564276224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828461.791472:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564276352:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828461.791491:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828461.791503:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828461/real 1787828461] req@ffff976882aec3c0 x1874670564276352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828461.791515:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564276352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828461.791522:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828461.791526:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828461.791529:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564276352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828462.046925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828462.046925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828462.047001:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828462.814943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828463.070933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828463.070960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828463.070982:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828463.839063:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828464.094902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828464.094904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828464.094979:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828464.863076:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828465.118955:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828465.118959:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828465.119082:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828465.887060:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828466.142937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828466.142960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828466.143077:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828466.910981:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828466.910991:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828466.911010:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828466.911022:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3406 00000100:00080000:1.0:1787828466.911047:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828466.911055:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828466.911056:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828466.911111:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564276480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828466.911148:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828466.911151:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828466.911153:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828466.911157:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3406 00000100:00080000:1.0:1787828466.911162:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828466.911166:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828466.911177:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828466.911187:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564276480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828466.911191:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564276608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828466.911214:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828466.911218:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828466.911220:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828466.911225:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3406 00000100:00080000:1.0:1787828466.911230:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828466.911233:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828466.911237:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828466.911246:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564276736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828466.911255:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828466.911258:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828466.911259:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828466.911263:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3406 00000100:00080000:1.0:1787828466.911267:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828466.911289:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:3.0:1787828466.911292:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787828466.911298:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828466.911312:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564276864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828466.911327:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828466.911330:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828466.911331:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828466.911335:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3406 00000100:00080000:1.0:1787828466.911355:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828466.911360:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828466.911366:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828466.911367:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828466/real 1787828466] req@ffff976889efe940 x1874670564276480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828466.911387:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564276992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828466.911398:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564276480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828466.911423:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828466.911448:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828466.911453:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564276480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828466.911476:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564276608:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828466.911514:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828466.911526:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828466/real 1787828466] req@ffff976889eff0c0 x1874670564276608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828466.911537:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564276608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828466.911554:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828466.911557:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828466.911559:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564276608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828466.911582:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564276736:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828466.911594:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828466.911600:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828466/real 1787828466] req@ffff976889effc00 x1874670564276736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828466.911617:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564276736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828466.911623:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828466.911624:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828466.911626:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564276736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828466.911647:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564276864:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828466.911662:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828466.911670:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828466/real 1787828466] req@ffff976889efed00 x1874670564276864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828466.911685:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564276864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828466.911692:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828466.911694:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828466.911696:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564276864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828466.911701:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564276992:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828466.911714:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828466.911721:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828466/real 1787828466] req@ffff976889efc3c0 x1874670564276992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828466.911729:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564276992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828466.911736:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828466.911738:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828466.911740:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564276992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828467.166953:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828467.167071:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828467.167075:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828467.934964:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828468.190918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828468.190955:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828468.191014:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828468.958893:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828469.214912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828469.214925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828469.215013:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828469.982938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828470.238968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828470.238968:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828470.239046:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828471.006902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828471.262935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828471.262946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828471.263150:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828472.030976:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828472.030989:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1787828472.030995:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828472.030996:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828472.031010:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3411 00000100:00080000:1.0:1787828472.031025:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828472.031035:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828472.031090:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564277120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828472.031119:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828472.031122:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828472.031124:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828472.031130:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3411 00000100:00080000:1.0:1787828472.031132:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828472.031136:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828472.031148:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828472.031162:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564277248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828472.031172:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828472.031175:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828472.031176:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828472.031180:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3411 00000100:00080000:1.0:1787828472.031183:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828472.031191:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828472.031191:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564277120:192.168.203.139@tcp:250:kworker.0 00000008:00100000:1.0:1787828472.031211:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828472.031224:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564277376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828472.031233:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828472.031236:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828472.031238:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828472.031242:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3411 00000100:00080000:1.0:1787828472.031247:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828472.031254:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828472.031261:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828472.031283:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564277504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828472.031297:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828472.031298:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787828472.031303:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828472.031305:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828472.031310:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3411 00000100:00080000:1.0:1787828472.031319:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828472.031325:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828472.031332:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828472.031347:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564277632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828472.031384:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828472/real 1787828472] req@ffff976889efc3c0 x1874670564277120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828472.031411:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564277120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828472.031427:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828472.031433:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828472.031438:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564277120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828472.031471:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564277248:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828472.031514:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828472.031531:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828472/real 1787828472] req@ffff976889efed00 x1874670564277248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828472.031547:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564277248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828472.031558:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828472.031562:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828472.031566:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564277248:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828472.031574:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564277376:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828472.031595:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828472.031607:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828472/real 1787828472] req@ffff976889effc00 x1874670564277376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828472.031621:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564277376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828472.031639:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828472.031644:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828472.031647:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564277376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828472.031654:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564277504:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828472.031673:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828472.031684:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828472/real 1787828472] req@ffff976889eff0c0 x1874670564277504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828472.031698:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564277504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828472.031708:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828472.031712:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828472.031715:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564277504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828472.031722:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564277632:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828472.031741:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828472.031752:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828472/real 1787828472] req@ffff976889efe940 x1874670564277632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828472.031765:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564277632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828472.031775:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828472.031779:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828472.031783:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564277632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828472.286941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828472.286941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828472.287044:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828473.054923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828473.310979:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828473.311038:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828473.311099:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828474.078960:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828474.334941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828474.334941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828474.335128:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828475.102932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828475.358975:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828475.358975:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828475.359112:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828476.126911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828476.382918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828476.382918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828476.383043:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828477.150916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828477.151024:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828477.151034:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828477.151039:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828477.151049:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3417 00000100:00080000:2.0:1787828477.151060:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828477.151067:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828477.151109:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564277760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828477.151150:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828477.151154:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828477.151155:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828477.151160:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3417 00000100:00080000:2.0:1787828477.151164:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828477.151168:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828477.151180:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828477.151191:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564277888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828477.151202:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828477.151206:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828477.151209:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828477.151214:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564277760:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828477.151218:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3417 00000100:00080000:2.0:1787828477.151231:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828477.151237:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828477.151241:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828477.151252:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564278016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828477.151261:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828477.151265:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828477.151267:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828477.151271:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3417 00000100:00080000:2.0:1787828477.151274:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828477.151277:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828477.151281:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828477.151290:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564278144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828477.151299:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828477.151301:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828477.151303:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828477.151306:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3417 00000100:00080000:2.0:1787828477.151311:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828477.151314:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828477.151317:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828477.151325:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564278272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828477.151341:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828477.151401:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828477/real 1787828477] req@ffff976889cfa940 x1874670564277760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828477.151421:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564277760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828477.151433:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828477.151437:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828477.151441:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564277760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828477.151471:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564277888:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828477.151495:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828477.151506:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828477/real 1787828477] req@ffff976889cfbc00 x1874670564277888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828477.151516:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564277888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828477.151523:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828477.151526:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828477.151530:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564277888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828477.151536:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564278016:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828477.151551:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828477.151558:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828477/real 1787828477] req@ffff976889cfa580 x1874670564278016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828477.151567:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564278016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828477.151574:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828477.151576:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828477.151578:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564278016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828477.151583:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564278144:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828477.151597:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828477.151605:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828477/real 1787828477] req@ffff976889cf8000 x1874670564278144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828477.151614:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564278144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828477.151620:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828477.151623:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828477.151625:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564278144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828477.151630:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564278272:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828477.151642:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828477.151650:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828477/real 1787828477] req@ffff976889cfb0c0 x1874670564278272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828477.151659:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564278272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828477.151666:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828477.151668:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828477.151670:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564278272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828477.407008:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828477.407113:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828477.407160:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828478.174990:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828478.430988:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828478.430990:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828478.431069:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828479.198938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828479.454958:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828479.454962:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828479.454982:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828480.223042:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828480.478951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828480.478990:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828480.479056:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828481.247159:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828481.502944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828481.503043:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828481.503126:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828482.270945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828482.271127:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828482.271134:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828482.271138:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828482.271147:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3422 00000100:00080000:1.0:1787828482.271158:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828482.271166:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828482.271204:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564278400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828482.271229:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828482.271232:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828482.271234:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828482.271237:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3422 00000100:00080000:1.0:1787828482.271241:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828482.271245:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828482.271257:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828482.271267:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564278400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828482.271274:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564278528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828482.271288:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828482.271290:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828482.271292:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828482.271295:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3422 00000100:00080000:1.0:1787828482.271299:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828482.271301:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828482.271323:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828482.271352:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564278656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828482.271365:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828482.271374:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787828482.271376:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828482.271383:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828482.271389:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3422 00000100:00080000:1.0:1787828482.271396:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828482.271402:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828482.271416:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828482.271427:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564278784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828482.271452:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828482.271454:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828482.271456:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828482.271458:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3422 00000100:00080000:1.0:1787828482.271462:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828482.271467:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828482.271472:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828482/real 1787828482] req@ffff976889efcb40 x1874670564278400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828482.271473:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828482.271493:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564278912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828482.271512:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564278400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828482.271524:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828482.271533:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828482.271539:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564278400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828482.271593:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564278528:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828482.271623:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828482.271634:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828482/real 1787828482] req@ffff976889efe1c0 x1874670564278528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828482.271649:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564278528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828482.271656:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828482.271658:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828482.271661:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564278528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828482.271665:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564278656:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828482.271677:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828482.271684:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828482/real 1787828482] req@ffff976889efe580 x1874670564278656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828482.271708:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564278656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828482.271714:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828482.271715:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828482.271717:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564278656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828482.271721:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564278784:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828482.271732:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828482.271738:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828482/real 1787828482] req@ffff976889eff480 x1874670564278784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828482.271745:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564278784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828482.271755:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828482.271772:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828482.271774:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564278784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828482.271778:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564278912:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828482.271797:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828482.271804:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828482/real 1787828482] req@ffff976889efc780 x1874670564278912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828482.271811:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564278912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828482.271817:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828482.271819:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828482.271821:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564278912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828482.526940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828482.526973:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828482.527036:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828483.294978:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828483.550965:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828483.551003:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828483.551005:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828484.318940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828484.574961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828484.574961:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828484.575035:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828485.342978:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828485.598912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828485.599001:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828485.599026:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828486.366992:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828486.623005:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828486.623025:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828486.623120:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828487.391009:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828487.391021:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1787828487.391021:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828487.391026:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828487.391036:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3427 00000100:00080000:1.0:1787828487.391047:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828487.391053:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828487.391091:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564279040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828487.391120:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828487.391130:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828487.391132:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828487.391141:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3427 00000100:00080000:1.0:1787828487.391145:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828487.391148:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828487.391158:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828487.391178:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564279040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828487.391179:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564279168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828487.391224:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828487.391229:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828487.391232:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828487.391238:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3427 00000100:00080000:1.0:1787828487.391244:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828487.391248:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828487.391255:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828487.391273:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564279296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828487.391284:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828487.391293:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1787828487.391295:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787828487.391297:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828487.391307:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3427 00000100:00080000:1.0:1787828487.391313:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828487.391319:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828487.391326:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828487.391346:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564279424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828487.391356:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828487.391361:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828487/real 1787828487] req@ffff976889efc780 x1874670564279040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828487.391362:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828487.391371:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828487.391390:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3427 00000100:00080000:1.0:1787828487.391397:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828487.391401:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564279040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828487.391410:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:0.0:1787828487.391420:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787828487.391424:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828487.391428:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828487.391435:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564279040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828487.391444:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564279552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828487.391469:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564279168:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828487.391503:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828487.391528:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828487/real 1787828487] req@ffff976889eff480 x1874670564279168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828487.391540:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564279168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828487.391555:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828487.391562:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828487.391572:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564279168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828487.391585:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564279296:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828487.391607:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828487.391617:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828487/real 1787828487] req@ffff976889efe580 x1874670564279296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828487.391626:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564279296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828487.391632:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828487.391634:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828487.391636:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564279296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828487.391641:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564279424:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828487.391666:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828487.391673:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828487/real 1787828487] req@ffff976889efe1c0 x1874670564279424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828487.391682:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564279424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828487.391688:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828487.391690:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828487.391692:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564279424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828487.391696:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564279552:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828487.391708:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828487.391715:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828487/real 1787828487] req@ffff976889efcb40 x1874670564279552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828487.391732:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564279552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828487.391738:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828487.391740:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828487.391742:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564279552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828487.646963:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828487.647004:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828487.647071:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828488.415003:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828488.670998:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828488.670998:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828488.671091:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828489.438962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828489.694965:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828489.695027:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828489.695112:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828490.462951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828490.718930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828490.719020:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828490.719887:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828491.486931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828491.742953:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828491.742972:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828491.743035:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828492.510981:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828492.510985:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828492.510999:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828492.511005:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828492.511021:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3432 00000100:00080000:3.0:1787828492.511036:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828492.511047:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828492.511095:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564279680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828492.511127:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828492.511131:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828492.511133:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828492.511137:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3432 00000100:00080000:3.0:1787828492.511142:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828492.511146:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828492.511157:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828492.511180:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564279808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828492.511194:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564279680:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787828492.511196:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828492.511208:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828492.511210:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828492.511215:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3432 00000100:00080000:3.0:1787828492.511218:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828492.511222:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828492.511227:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828492.511239:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564279936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828492.511250:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828492.511254:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828492.511256:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828492.511261:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3432 00000100:00080000:3.0:1787828492.511265:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828492.511269:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828492.511275:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1787828492.511282:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828492.511293:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564280064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828492.511304:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828492.511307:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828492.511309:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828492.511313:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3432 00000100:00080000:3.0:1787828492.511316:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828492.511320:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828492.511327:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828492.511337:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828492/real 1787828492] req@ffff97688cb5c780 x1874670564279680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828492.511349:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564280192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828492.511365:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564279680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828492.511381:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828492.511389:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828492.511397:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564279680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828492.511430:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564279808:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828492.511487:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828492.511508:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828492/real 1787828492] req@ffff97688cb5e1c0 x1874670564279808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828492.511522:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564279808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828492.511547:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828492.511551:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828492.511555:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564279808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828492.511563:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564279936:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828492.511603:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828492.511615:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828492/real 1787828492] req@ffff97688cb5d2c0 x1874670564279936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828492.511627:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564279936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828492.511636:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828492.511639:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828492.511642:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564279936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828492.511647:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564280064:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828492.511685:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828492.511696:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828492/real 1787828492] req@ffff97688cb5f0c0 x1874670564280064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828492.511708:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564280064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828492.511717:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828492.511720:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828492.511724:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564280064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828492.511746:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564280192:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828492.511767:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828492.511778:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828492/real 1787828492] req@ffff97688cb5d680 x1874670564280192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828492.511791:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564280192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828492.511801:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828492.511804:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828492.511808:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564280192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828492.766907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828492.766934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828492.766978:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828493.534931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828493.790903:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828493.790903:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828493.790941:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828494.558930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828494.814913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828494.814960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828494.814995:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828495.582920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828495.838900:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828495.838906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828495.839033:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828496.606959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828496.862960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828496.862961:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828496.863034:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828497.630986:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828497.630995:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828497.631002:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828497.631013:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828497.631029:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3437 00000100:00080000:0.0:1787828497.631043:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828497.631053:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828497.631104:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564280320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828497.631148:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828497.631153:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828497.631155:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828497.631160:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3437 00000100:00080000:0.0:1787828497.631166:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828497.631173:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828497.631188:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828497.631218:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564280448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828497.631242:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828497.631250:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564280320:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828497.631253:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828497.631262:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828497.631275:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3437 00000100:00080000:0.0:1787828497.631283:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828497.631291:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828497.631300:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828497.631323:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564280576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828497.631337:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828497.631342:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828497.631345:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828497.631351:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3437 00000100:00080000:0.0:1787828497.631357:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787828497.631368:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828497.631369:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828497.631384:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828497.631399:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564280704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828497.631413:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828497.631416:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828497.631419:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828497.631424:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3437 00000100:00080000:0.0:1787828497.631429:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828497.631435:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828497.631442:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828497.631449:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828497/real 1787828497] req@ffff976882aef0c0 x1874670564280320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828497.631462:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564280832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828497.631500:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564280320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828497.631520:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828497.631528:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828497.631535:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564280320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828497.631577:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564280448:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828497.631626:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828497.631651:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828497/real 1787828497] req@ffff976882aec3c0 x1874670564280448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828497.631669:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564280448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828497.631681:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828497.631685:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828497.631690:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564280448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828497.631698:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564280576:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828497.631724:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828497.631737:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828497/real 1787828497] req@ffff976882aee580 x1874670564280576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828497.631752:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564280576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828497.631762:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828497.631766:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828497.631770:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564280576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828497.631779:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564280704:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828497.631872:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828497.631889:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828497/real 1787828497] req@ffff976882aeed00 x1874670564280704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828497.631905:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564280704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828497.631916:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828497.631920:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828497.631924:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564280704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828497.631932:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564280832:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828497.631953:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828497.631964:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828497/real 1787828497] req@ffff976882aee940 x1874670564280832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828497.631978:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564280832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828497.631987:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828497.631990:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828497.631994:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564280832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828497.886979:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828497.886990:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828497.887096:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828498.654955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828498.910968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828498.910996:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828498.911062:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828499.678951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828499.934943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828499.934943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828499.935051:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828500.702921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828500.958967:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828500.959020:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828500.959058:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828501.726971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828501.982997:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828501.982997:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828501.983216:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828502.750933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828502.750946:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828502.750958:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828502.750965:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828502.750979:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3442 00000100:00080000:0.0:1787828502.750993:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828502.751031:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828502.751096:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564280960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828502.751141:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828502.751147:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828502.751150:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828502.751158:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3442 00000100:00080000:0.0:1787828502.751165:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828502.751174:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828502.751189:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564280960:192.168.203.139@tcp:250:kworker.0 00000008:00100000:0.0:1787828502.751190:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828502.751224:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564281088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828502.751246:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828502.751262:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828502.751266:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828502.751274:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3442 00000100:00080000:0.0:1787828502.751298:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828502.751304:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828502.751312:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828502.751336:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828502.751344:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564281216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828502.751380:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828502.751386:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828502.751401:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828502.751409:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3442 00000100:00080000:0.0:1787828502.751418:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828502.751427:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828502.751435:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828502/real 1787828502] req@ffff976882aed680 x1874670564280960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828502.751440:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828502.751473:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564281344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828502.751493:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828502.751495:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564280960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828502.751519:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828502.751528:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828502.751531:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828502.751539:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3442 00000100:00080000:3.0:1787828502.751547:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:0.0:1787828502.751550:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828502.751562:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564280960:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828502.751565:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828502.751583:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828502.751621:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564281472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828502.751627:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564281088:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828502.751700:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828502.751718:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828502/real 1787828502] req@ffff976882aee940 x1874670564281088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828502.751733:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564281088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828502.751744:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828502.751771:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828502.751787:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564281088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828502.751795:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564281216:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828502.751852:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828502.751868:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828502/real 1787828502] req@ffff976882aeed00 x1874670564281216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828502.751881:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564281216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828502.751891:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828502.751894:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828502.751898:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564281216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828502.751904:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564281344:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828502.751926:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828502.751936:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828502/real 1787828502] req@ffff976882aee580 x1874670564281344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828502.751949:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564281344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828502.751958:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828502.751961:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828502.751964:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564281344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828502.751985:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564281472:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828502.752017:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828502.752026:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828502/real 1787828502] req@ffff976882aec3c0 x1874670564281472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828502.752042:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564281472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828502.752048:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828502.752051:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828502.752053:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564281472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828503.006966:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828503.006968:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828503.007050:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828503.774916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828504.031011:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828504.031075:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828504.031167:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828504.799015:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828505.054900:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828505.054950:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828505.055025:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828505.822995:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828506.078943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828506.078951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828506.079049:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828506.846924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828507.102924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828507.102924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828507.103054:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828507.870901:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828507.870912:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828507.870916:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828507.870956:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3447 00000100:00080000:0.0:1787828507.870966:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828507.870972:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828507.870992:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828507.871007:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564281600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828507.871034:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828507.871038:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828507.871039:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828507.871042:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3447 00000100:00080000:0.0:1787828507.871044:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828507.871047:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828507.871055:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828507.871065:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564281728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828507.871073:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828507.871076:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564281600:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828507.871077:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828507.871081:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828507.871087:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3447 00000100:00080000:0.0:1787828507.871089:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828507.871091:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828507.871094:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828507.871103:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564281856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828507.871109:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828507.871112:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828507.871113:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828507.871118:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3447 00000100:00080000:0.0:1787828507.871121:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828507.871126:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828507.871131:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828507.871143:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564281984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828507.871153:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828507.871157:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828507.871160:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828507.871165:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3447 00000100:00080000:0.0:1787828507.871171:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828507.871177:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828507.871184:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828507.871188:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564281728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828507.871197:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564282112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828507.871234:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564281856:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828507.871275:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564281984:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828507.871296:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564282112:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828508.126965:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828508.126994:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828508.127132:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828508.895102:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828509.151099:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828509.151124:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828509.151178:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828509.919072:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828510.174944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828510.174947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828510.175151:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828510.943242:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:0.0:1787828510.943281:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:0.0:1787828510.943337:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828510.943357:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828510.943401:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828510.943424:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828510.943439:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828510.943456:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:2.0:1787828510.943458:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828507/real 1787828510] req@ffff976882aec3c0 x1874670564281728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828510.943492:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564281728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828510.943518:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828510.943525:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828510.943531:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564281728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828510.943541:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828507/real 1787828510] req@ffff976882aee580 x1874670564281856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828510.943558:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564281856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828510.943569:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828510.943574:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828510.943578:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564281856:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828510.943587:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828507/real 1787828510] req@ffff976882aeed00 x1874670564281984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828510.943603:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564281984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828510.943614:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828510.943619:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828510.943623:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564281984:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828510.943631:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828507/real 1787828510] req@ffff976882aef0c0 x1874670564281600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828510.943646:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564281600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828510.943657:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828510.943661:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828510.943665:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564281600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828510.943720:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828507/real 1787828510] req@ffff976882aee940 x1874670564282112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828510.943739:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564282112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828510.943749:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828510.943753:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828510.943757:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564282112:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828511.198932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828511.198932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828511.199013:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828511.966988:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828512.222924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828512.222924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828512.223013:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828512.991022:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828512.991038:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828512.991044:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828512.991058:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3452 00000100:00100000:3.0:1787828512.991060:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828512.991073:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828512.991086:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828512.991134:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564282240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828512.991177:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828512.991182:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828512.991186:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828512.991191:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3452 00000100:00080000:2.0:1787828512.991196:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828512.991201:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828512.991215:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828512.991235:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564282240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828512.991238:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564282368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828512.991262:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828512.991267:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828512.991269:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828512.991275:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3452 00000100:00080000:2.0:1787828512.991279:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828512.991284:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828512.991291:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828512.991306:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564282496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828512.991320:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828512.991327:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828512.991333:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787828512.991334:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828512.991351:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3452 00000100:00080000:2.0:1787828512.991362:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828512.991371:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828512.991382:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828512.991415:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828512/real 1787828512] req@ffff976889cf8f00 x1874670564282240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828512.991417:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564282624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828512.991491:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828512.991498:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828512.991502:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828512.991509:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564282240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828512.991517:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3452 00000100:00080000:2.0:1787828512.991533:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828512.991550:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:3.0:1787828512.991553:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787828512.991563:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787828512.991569:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828512.991583:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564282240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828512.991598:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564282752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828512.991637:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564282368:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828512.991724:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828512.991751:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828512/real 1787828512] req@ffff976889cfb0c0 x1874670564282368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828512.991779:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564282368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828512.991797:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828512.991804:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828512.991811:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564282368:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828512.991867:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564282496:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828512.991913:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828512.991938:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828512/real 1787828512] req@ffff976889cf8000 x1874670564282496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828512.991964:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564282496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828512.991982:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828512.991989:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828512.991996:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564282496:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828512.992008:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564282624:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828512.992051:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828512.992076:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828512/real 1787828512] req@ffff976889cfa580 x1874670564282624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828512.992103:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564282624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828512.992120:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828512.992126:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828512.992133:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564282624:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828512.992146:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564282752:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828512.992188:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828512.992210:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828512/real 1787828512] req@ffff976889cfbc00 x1874670564282752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828512.992866:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564282752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828512.992888:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828512.992895:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828512.992902:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564282752:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828513.246949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828513.246954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828513.247110:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828514.015131:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828514.270992:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828514.271034:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828514.271058:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828515.038948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828515.294922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828515.294963:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828515.294993:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828516.062927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828516.318919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828516.318919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828516.318977:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828517.086968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828517.342944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828517.343026:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828517.343089:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828518.110957:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828518.110962:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828518.110975:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828518.110981:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828518.110993:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3458 00000100:00080000:1.0:1787828518.111007:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828518.111016:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828518.111073:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564282880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828518.111131:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828518.111135:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828518.111138:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828518.111143:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3458 00000100:00080000:1.0:1787828518.111148:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828518.111153:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828518.111164:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828518.111179:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564282880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828518.111202:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564283008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828518.111220:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828518.111225:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828518.111228:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828518.111232:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3458 00000100:00080000:1.0:1787828518.111237:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828518.111242:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828518.111248:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828518.111263:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564283136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828518.111274:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828518.111277:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828518.111279:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828518.111284:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3458 00000100:00080000:1.0:1787828518.111289:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828518.111293:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:3.0:1787828518.111303:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787828518.111305:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828518.111326:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564283264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828518.111344:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828518.111349:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828518.111353:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828518.111362:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828518/real 1787828518] req@ffff976889efe940 x1874670564282880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828518.111365:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3458 00000100:00080000:1.0:1787828518.111380:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828518.111401:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828518.111407:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564282880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828518.111414:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787828518.111426:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828518.111434:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828518.111445:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564282880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828518.111447:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564283392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828518.111501:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564283008:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828518.111551:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828518.111568:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828518/real 1787828518] req@ffff976889eff0c0 x1874670564283008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828518.111584:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564283008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828518.111596:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828518.111600:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828518.111604:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564283008:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828518.111611:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564283136:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828518.111631:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828518.111673:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828518/real 1787828518] req@ffff976889effc00 x1874670564283136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828518.111691:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564283136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828518.111710:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828518.111714:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828518.111718:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564283136:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828518.111726:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564283264:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828518.111759:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828518.111772:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828518/real 1787828518] req@ffff976889efed00 x1874670564283264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828518.111787:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564283264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828518.111797:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828518.111803:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828518.111810:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564283264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828518.111821:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564283392:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828518.111904:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828518.111925:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828518/real 1787828518] req@ffff976889efc3c0 x1874670564283392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828518.111942:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564283392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828518.111952:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828518.111956:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828518.111961:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564283392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828518.366930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828518.366930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828518.367003:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828519.134906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828519.390925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828519.390925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828519.390971:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828520.158902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828520.414943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828520.414945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828520.415170:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828521.182940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828521.438891:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828521.438895:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828521.438968:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828522.206991:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828522.462966:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828522.462978:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828522.463111:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828523.230987:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828523.231094:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828523.231116:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828523.231125:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828523.231143:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3463 00000100:00080000:3.0:1787828523.231160:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828523.231175:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828523.231241:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564283520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828523.231289:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828523.231295:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828523.231298:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828523.231303:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3463 00000100:00080000:3.0:1787828523.231308:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828523.231314:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828523.231327:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828523.231347:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564283648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828523.231363:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828523.231368:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828523.231376:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828523.231384:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3463 00000100:00080000:3.0:1787828523.231390:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828523.231396:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828523.231406:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828523.231410:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564283520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828523.231426:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564283776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828523.231440:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828523.231444:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828523.231446:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828523.231456:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3463 00000100:00080000:3.0:1787828523.231461:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828523.231468:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828523.231477:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828523.231495:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564283904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828523.231507:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828523.231512:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828523.231514:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828523.231523:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3463 00000800:00000100:1.0:1787828523.231526:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787828523.231530:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828523.231537:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828523.231544:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828523.231563:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564284032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828523.231644:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828523/real 1787828523] req@ffff97688cb5f840 x1874670564283520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828523.231670:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564283520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828523.231693:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828523.231698:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828523.231703:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564283520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828523.231716:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564283648:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828523.231747:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828523.231768:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828523/real 1787828523] req@ffff97688cb5de00 x1874670564283648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828523.231782:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564283648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828523.231791:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828523.231794:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828523.231798:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564283648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828523.231889:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564283776:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828523.231934:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828523.231955:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828523/real 1787828523] req@ffff97688cb5cb40 x1874670564283776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828523.231971:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564283776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828523.231982:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828523.231987:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828523.231993:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564283776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828523.232004:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564283904:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828523.232036:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828523.232057:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828523/real 1787828523] req@ffff97688cb5c000 x1874670564283904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828523.232081:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564283904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828523.232096:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828523.232103:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828523.232109:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564283904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828523.232119:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564284032:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828523.232153:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828523.232172:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828523/real 1787828523] req@ffff97688cb5d680 x1874670564284032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828523.232188:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564284032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828523.232197:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828523.232201:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828523.232204:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564284032:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828523.486990:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828523.486990:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828523.487088:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828524.254928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828524.510917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828524.510938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828524.510990:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828525.278904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828525.534948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828525.534953:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828525.535140:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828526.302902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828526.558953:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828526.558956:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828526.559052:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828527.326944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828527.582915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828527.582926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828527.583053:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828528.350994:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828528.351002:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828528.351011:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828528.351017:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828528.351028:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3468 00000100:00080000:0.0:1787828528.351040:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828528.351048:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828528.351089:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564284160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828528.351120:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828528.351125:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828528.351127:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828528.351131:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3468 00000100:00080000:0.0:1787828528.351135:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828528.351138:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828528.351156:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828528.351174:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564284288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828528.351181:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564284160:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828528.351193:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828528.351198:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828528.351200:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828528.351204:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3468 00000100:00080000:0.0:1787828528.351207:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828528.351211:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828528.351217:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828528.351230:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564284416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828528.351241:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828528.351245:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828528.351247:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828528.351251:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3468 00000100:00080000:0.0:1787828528.351264:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828528.351267:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828528.351274:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1787828528.351277:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828528.351290:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564284544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828528.351301:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828528.351312:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828528.351315:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828528.351319:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3468 00000100:00080000:0.0:1787828528.351332:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828528.351336:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828528.351342:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828528.351354:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564284672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828528.351369:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828528/real 1787828528] req@ffff976882aed680 x1874670564284160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828528.351403:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564284160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828528.351416:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828528.351420:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828528.351436:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564284160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828528.351464:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564284288:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828528.351493:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828528.351508:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828528/real 1787828528] req@ffff976882aee940 x1874670564284288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828528.351521:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564284288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828528.351531:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828528.351534:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828528.351538:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564284288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828528.351544:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564284416:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828528.351560:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828528.351594:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828528/real 1787828528] req@ffff976882aef0c0 x1874670564284416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828528.351617:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564284416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828528.351631:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828528.351636:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828528.351641:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564284416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828528.351651:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564284544:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828528.351686:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828528.351705:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828528/real 1787828528] req@ffff976882aeed00 x1874670564284544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828528.351720:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564284544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828528.351729:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828528.351733:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828528.351736:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564284544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828528.351742:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564284672:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828528.351764:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828528.351774:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828528/real 1787828528] req@ffff976882aee580 x1874670564284672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828528.351785:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564284672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828528.351800:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828528.351803:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828528.351807:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564284672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828528.606996:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828528.606998:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828528.607151:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828529.374943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828529.630952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828529.630967:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828529.631022:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828530.398980:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828530.654940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828530.654940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828530.655130:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828531.422935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828531.679027:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828531.679126:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828531.679731:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828532.447023:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828532.702921:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828532.702921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828532.702987:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828533.470949:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828533.470967:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1787828533.470968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828533.470974:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828533.470991:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3473 00000100:00080000:0.0:1787828533.471007:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828533.471036:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828533.471089:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564284800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828533.471140:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828533.471148:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828533.471151:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828533.471157:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3473 00000100:00080000:0.0:1787828533.471163:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828533.471170:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828533.471183:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828533.471210:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564284800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828533.471211:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564284928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828533.471252:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828533.471256:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828533.471258:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828533.471263:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3473 00000100:00080000:0.0:1787828533.471267:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828533.471272:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828533.471277:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828533.471291:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564285056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828533.471301:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828533.471305:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828533.471307:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828533.471314:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828533.471318:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3473 00000100:00080000:0.0:1787828533.471325:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828533.471332:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828533.471342:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828533.471363:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564285184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828533.471369:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828533/real 1787828533] req@ffff976882aec3c0 x1874670564284800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828533.471423:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828533.471430:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564284800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828533.471431:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828533.471437:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828533.471450:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3473 00000100:00080000:1.0:1787828533.471460:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828533.471461:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828533.471490:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:1.0:1787828533.471492:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000008:00100000:0.0:1787828533.471502:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828533.471503:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564284800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828533.471529:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564285312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828533.471568:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564284928:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828533.471622:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828533.471640:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828533/real 1787828533] req@ffff976882aee580 x1874670564284928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828533.471661:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564284928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828533.471670:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828533.471674:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828533.471677:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564284928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828533.471683:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564285056:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828533.471700:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828533.471709:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828533/real 1787828533] req@ffff976882aeed00 x1874670564285056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828533.471721:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564285056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828533.471728:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828533.471731:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828533.471734:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564285056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828533.471739:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564285184:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828533.471755:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828533.471800:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828533/real 1787828533] req@ffff976882aef0c0 x1874670564285184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828533.471867:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564285184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828533.471888:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828533.471892:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828533.471896:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564285184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828533.471949:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564285312:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828533.471976:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828533.472012:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828533/real 1787828533] req@ffff976882aee940 x1874670564285312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828533.472034:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564285312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828533.472050:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828533.472056:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828533.472062:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564285312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828533.726940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828533.726940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828533.726958:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828534.494917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828534.750951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828534.750956:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828534.750984:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828535.518917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828535.774927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828535.774997:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828535.775040:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828536.542920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828536.798943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828536.799001:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828536.799047:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828537.566944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828537.822917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828537.822974:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828537.823065:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828538.590964:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828538.591007:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828538.591020:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828538.591026:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828538.591038:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3478 00000100:00080000:0.0:1787828538.591052:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828538.591063:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828538.591112:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564285440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828538.591157:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828538.591162:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828538.591165:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828538.591170:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3478 00000100:00080000:0.0:1787828538.591174:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828538.591180:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828538.591192:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828538.591210:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564285568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828538.591217:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564285440:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828538.591248:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828538.591253:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828538.591256:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828538.591261:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3478 00000100:00080000:0.0:1787828538.591267:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828538.591271:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828538.591277:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828538.591294:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564285696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828538.591306:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828538.591310:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828538.591313:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828538.591318:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3478 00000100:00080000:0.0:1787828538.591325:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828538.591330:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828538.591337:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828538.591347:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828538.591374:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564285824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828538.591387:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828538.591391:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828538.591394:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828538.591400:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3478 00000100:00080000:0.0:1787828538.591406:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828538.591408:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828538/real 1787828538] req@ffff976882aed680 x1874670564285440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828538.591416:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828538.591428:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828538.591456:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564285440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828538.591458:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564285952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828538.591545:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828538.591556:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828538.591563:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564285440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828538.591597:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564285568:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828538.591651:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828538.591681:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828538/real 1787828538] req@ffff976882aee940 x1874670564285568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828538.591708:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564285568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828538.591720:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828538.591724:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828538.591728:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564285568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828538.591746:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564285696:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828538.591775:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828538.591789:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828538/real 1787828538] req@ffff976882aef0c0 x1874670564285696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828538.591812:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564285696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828538.591853:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828538.591858:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828538.591862:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564285696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828538.591870:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564285824:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828538.591899:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828538.591913:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828538/real 1787828538] req@ffff976882aeed00 x1874670564285824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828538.591928:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564285824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828538.591938:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828538.591942:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828538.591945:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564285824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828538.591952:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564285952:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828538.591970:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828538.591981:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828538/real 1787828538] req@ffff976882aee580 x1874670564285952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828538.591995:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564285952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828538.592021:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828538.592024:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828538.592028:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564285952:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828538.846981:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828538.846987:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828538.847304:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828539.614902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828539.870944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828539.870948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828539.870963:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828540.638940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828540.894978:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828540.894979:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828540.895008:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828541.662978:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828541.918927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828541.918928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828541.918987:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828542.686976:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828542.943051:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828542.943055:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828542.943166:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828543.710947:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828543.710958:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828543.710963:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828543.710978:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3483 00000100:00100000:1.0:1787828543.710986:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828543.710995:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828543.711012:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828543.711065:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564286080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828543.711108:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828543.711113:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828543.711115:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828543.711120:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3483 00000100:00080000:0.0:1787828543.711126:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828543.711133:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828543.711146:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828543.711156:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564286080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828543.711165:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564286208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828543.711175:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828543.711180:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828543.711182:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828543.711188:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3483 00000100:00080000:0.0:1787828543.711192:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828543.711195:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828543.711200:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828543.711212:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564286336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828543.711220:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828543.711223:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828543.711225:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828543.711230:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3483 00000100:00080000:0.0:1787828543.711235:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828543.711245:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828543.711252:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828543.711258:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828543.711273:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564286464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828543.711288:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828543.711301:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828543.711304:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828543.711310:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3483 00000100:00080000:0.0:1787828543.711316:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828543.711322:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828543.711328:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828543.711335:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828543/real 1787828543] req@ffff976882aec3c0 x1874670564286080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828543.711344:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564286592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828543.711415:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564286080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828543.711433:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828543.711438:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828543.711443:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564286080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828543.711527:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564286208:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828543.711555:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828543.711570:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828543/real 1787828543] req@ffff976882aee580 x1874670564286208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828543.711582:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564286208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828543.711591:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828543.711594:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828543.711598:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564286208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828543.711604:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564286336:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828543.711622:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828543.711638:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828543/real 1787828543] req@ffff976882aeed00 x1874670564286336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828543.711656:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564286336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828543.711669:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828543.711674:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828543.711679:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564286336:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828543.711690:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564286464:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828543.711718:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828543.711730:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828543/real 1787828543] req@ffff976882aef0c0 x1874670564286464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828543.711742:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564286464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828543.711750:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828543.711753:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828543.711757:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564286464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828543.711763:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564286592:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828543.711778:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828543.711787:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828543/real 1787828543] req@ffff976882aee940 x1874670564286592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828543.711797:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564286592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828543.711804:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828543.711807:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828543.711810:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564286592:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828543.966915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828543.966978:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828543.966983:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828544.734975:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828544.990932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828544.990950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828544.990951:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828545.759004:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828546.014989:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828546.014999:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828546.015098:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828546.782927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828547.038966:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828547.039021:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828547.039331:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828547.806925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828548.062955:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828548.062959:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828548.063052:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828548.830945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828548.830967:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828548.830979:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828548.830985:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828548.830999:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3488 00000100:00080000:2.0:1787828548.831015:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828548.831023:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828548.831067:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564286720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828548.831100:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828548.831104:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828548.831107:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828548.831112:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3488 00000100:00080000:2.0:1787828548.831117:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828548.831121:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828548.831133:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828548.831142:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564286720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828548.831151:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564286848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828548.831165:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828548.831169:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828548.831172:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828548.831176:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3488 00000100:00080000:2.0:1787828548.831179:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828548.831183:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828548.831188:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828548.831202:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564286976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828548.831211:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828548.831215:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828548.831217:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828548.831227:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828548.831228:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3488 00000100:00080000:2.0:1787828548.831240:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828548.831248:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828548.831257:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828548.831279:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564287104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828548.831283:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828548/real 1787828548] req@ffff976889cfa940 x1874670564286720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828548.831332:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828548.831339:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564286720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828548.831340:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828548.831346:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828548.831357:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3488 00000100:00080000:2.0:1787828548.831369:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828548.831372:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828548.831377:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:1.0:1787828548.831383:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000008:00100000:2.0:1787828548.831389:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828548.831391:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564286720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828548.831413:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564287232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828548.831444:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564286848:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828548.831479:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828548.831494:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828548/real 1787828548] req@ffff976889cfad00 x1874670564286848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828548.831507:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564286848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828548.831517:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828548.831520:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828548.831523:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564286848:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828548.831530:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564286976:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828548.831546:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828548.831556:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828548/real 1787828548] req@ffff976889cf92c0 x1874670564286976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828548.831586:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564286976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828548.831600:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828548.831605:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828548.831610:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564286976:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828548.831620:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564287104:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828548.831652:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828548.831666:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828548/real 1787828548] req@ffff976889cf9e00 x1874670564287104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828548.831679:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564287104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828548.831692:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828548.831697:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828548.831702:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564287104:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828548.831728:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564287232:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828548.831761:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828548.831777:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828548/real 1787828548] req@ffff976889cfb480 x1874670564287232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828548.831790:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564287232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828548.831805:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828548.831808:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828548.831812:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564287232:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828549.086940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828549.086943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828549.087068:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828549.854894:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828550.110969:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828550.110969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828550.111002:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828550.879001:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828551.134953:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828551.134965:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828551.135080:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828551.903013:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828552.158954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828552.158959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828552.159045:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828552.926921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828553.182972:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828553.182987:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828553.183124:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828553.950961:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828553.950967:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828553.950979:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828553.950986:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828553.951012:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3493 00000100:00080000:2.0:1787828553.951029:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828553.951040:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828553.951100:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564287360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828553.951138:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828553.951158:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828553.951161:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828553.951165:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3493 00000100:00080000:2.0:1787828553.951168:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828553.951172:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828553.951182:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828553.951195:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564287488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828553.951204:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828553.951217:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828553.951218:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828553.951221:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3493 00000100:00080000:2.0:1787828553.951223:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828553.951226:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828553.951231:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828553.951238:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564287616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828553.951247:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828553.951252:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787828553.951254:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564287360:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828553.951254:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828553.951283:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3493 00000100:00080000:2.0:1787828553.951295:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828553.951300:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828553.951307:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828553.951326:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564287744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828553.951339:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828553.951343:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828553.951346:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828553.951350:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3493 00000100:00080000:2.0:1787828553.951355:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828553.951359:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828553.951364:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828553.951377:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564287872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828553.951474:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828553.951524:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828553/real 1787828553] req@ffff976889cfa1c0 x1874670564287360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828553.951541:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564287360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828553.951552:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828553.951556:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828553.951559:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564287360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828553.951566:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564287488:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828553.951587:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828553.951597:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828553/real 1787828553] req@ffff976889cfb480 x1874670564287488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828553.951613:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564287488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828553.951619:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828553.951622:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828553.951624:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564287488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828553.951666:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564287616:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828553.951683:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828553.951691:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828553/real 1787828553] req@ffff976889cf9e00 x1874670564287616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828553.951702:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564287616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828553.951709:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828553.951711:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828553.951713:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564287616:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828553.951718:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564287744:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828553.951732:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828553.951740:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828553/real 1787828553] req@ffff976889cf92c0 x1874670564287744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828553.951750:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564287744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828553.951763:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828553.951766:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828553.951768:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564287744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828553.951773:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564287872:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828553.951785:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828553.951792:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828553/real 1787828553] req@ffff976889cfad00 x1874670564287872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828553.951801:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564287872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828553.951809:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828553.951811:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828553.951813:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564287872:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828554.206971:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828554.206990:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828554.207090:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828554.974970:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828555.230974:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828555.230986:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828555.231062:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828555.998948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828556.254985:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828556.255031:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828556.255079:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828557.022898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828557.278896:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828557.278896:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828557.278967:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828558.046907:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828558.302936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828558.302972:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828558.303022:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828559.070925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828559.070966:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828559.070972:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828559.070976:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828559.070986:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3498 00000100:00080000:1.0:1787828559.070997:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828559.071004:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828559.071048:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564288000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828559.071082:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828559.071088:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828559.071093:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828559.071100:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3498 00000100:00080000:1.0:1787828559.071108:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828559.071114:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828559.071127:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828559.071143:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564288000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828559.071155:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564288128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828559.071170:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828559.071173:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828559.071175:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828559.071180:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3498 00000100:00080000:1.0:1787828559.071184:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828559.071189:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828559.071196:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828559.071236:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564288256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828559.071247:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828559.071251:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828559.071252:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828559.071258:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3498 00000100:00080000:1.0:1787828559.071266:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828559.071271:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828559.071278:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1787828559.071287:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828559.071301:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564288384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828559.071314:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828559.071318:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828559.071322:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828559.071329:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3498 00000100:00080000:1.0:1787828559.071332:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828559.071335:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828559.071343:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828559.071373:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564288512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828559.071384:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828559/real 1787828559] req@ffff976889efc3c0 x1874670564288000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828559.071420:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564288000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828559.071437:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828559.071444:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828559.071450:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564288000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828559.071486:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564288128:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828559.071522:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828559.071540:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828559/real 1787828559] req@ffff976889efed00 x1874670564288128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828559.071570:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564288128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828559.071582:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828559.071586:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828559.071592:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564288128:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828559.071621:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564288256:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828559.071650:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828559.071675:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828559/real 1787828559] req@ffff976889effc00 x1874670564288256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828559.071693:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564288256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828559.071709:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828559.071714:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828559.071718:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564288256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828559.071725:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564288384:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828559.071744:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828559.071754:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828559/real 1787828559] req@ffff976889eff0c0 x1874670564288384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828559.071765:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564288384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828559.071772:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828559.071775:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828559.071778:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564288384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828559.071782:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564288512:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828559.071796:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828559.071811:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828559/real 1787828559] req@ffff976889efe940 x1874670564288512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828559.071821:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564288512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828559.071828:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828559.071830:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828559.071833:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564288512:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828559.326894:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828559.326955:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828559.326972:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828560.094927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:1.0:1787828560.350968:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828560.351010:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828560.351056:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828561.118950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828561.374898:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828561.374899:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828561.374945:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828562.142893:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828562.398886:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828562.398936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828562.398956:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828563.166955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828563.423001:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828563.423001:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828563.423070:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828564.190947:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828564.190947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828564.190974:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828564.190978:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828564.190987:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3504 00000100:00080000:1.0:1787828564.190996:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828564.191003:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828564.191054:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564288640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828564.191077:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828564.191079:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828564.191096:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828564.191100:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3504 00000100:00080000:1.0:1787828564.191102:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828564.191105:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828564.191113:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828564.191121:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564288768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828564.191130:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828564.191133:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828564.191134:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828564.191136:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3504 00000100:00080000:1.0:1787828564.191138:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828564.191140:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828564.191144:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828564.191166:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564288640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828564.191168:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564288896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828564.191188:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828564.191190:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828564.191192:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828564.191196:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3504 00000100:00080000:1.0:1787828564.191198:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828564.191201:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828564.191205:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828564.191214:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564289024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828564.191220:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828564.191222:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828564.191223:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828564.191226:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3504 00000100:00080000:1.0:1787828564.191228:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828564.191232:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787828564.191235:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828564.191239:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828564.191246:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564289152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828564.191275:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828564/real 1787828564] req@ffff976889efcb40 x1874670564288640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828564.191288:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564288640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828564.191296:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828564.191299:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828564.191302:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564288640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828564.191307:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564288768:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828564.191340:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828564.191350:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828564/real 1787828564] req@ffff976889efe1c0 x1874670564288768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828564.191359:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564288768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828564.191365:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828564.191367:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828564.191369:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564288768:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828564.191387:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564288896:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828564.191399:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828564.191406:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828564/real 1787828564] req@ffff976889efe580 x1874670564288896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828564.191418:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564288896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828564.191423:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828564.191425:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828564.191427:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564288896:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828564.191431:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564289024:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828564.191441:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828564.191447:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828564/real 1787828564] req@ffff976889eff480 x1874670564289024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828564.191454:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564289024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828564.191459:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828564.191461:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828564.191463:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564289024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828564.191466:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564289152:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828564.191476:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828564.191481:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828564/real 1787828564] req@ffff976889efc780 x1874670564289152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828564.191488:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564289152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828564.191497:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828564.191499:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828564.191501:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564289152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828564.446936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828564.446965:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828564.447005:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828565.214976:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828565.470944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828565.470955:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828565.470987:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828566.238922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828566.494926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828566.494948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828566.495016:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828567.262943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828567.518937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828567.518962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828567.519005:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828568.286993:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828568.542945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828568.542945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828568.543050:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828569.310946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828569.311015:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828569.311044:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828569.311049:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828569.311061:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3509 00000100:00080000:2.0:1787828569.311074:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828569.311092:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828569.311135:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564289280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828569.311174:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828569.311178:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828569.311181:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828569.311186:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3509 00000100:00080000:2.0:1787828569.311191:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828569.311196:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828569.311208:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828569.311225:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564289408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828569.311235:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564289280:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828569.311268:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828569.311272:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828569.311274:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828569.311281:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3509 00000100:00080000:2.0:1787828569.311306:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828569.311311:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828569.311319:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828569.311357:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564289536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828569.311377:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828569.311384:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828569.311394:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828569.311399:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828569.311408:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3509 00000100:00080000:2.0:1787828569.311419:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828569.311429:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828569.311441:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828569.311448:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828569/real 1787828569] req@ffff976889cfa940 x1874670564289280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828569.311461:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564289664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828569.311563:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564289280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828569.311565:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828569.311594:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828569.311603:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828569.311604:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828569.311621:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828569.311628:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564289280:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828569.311640:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3509 00000100:00080000:2.0:1787828569.311651:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828569.311659:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828569.311666:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564289408:192.168.203.139@tcp:38:kworker.0 00000008:00100000:2.0:1787828569.311670:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828569.311695:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564289792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828569.311696:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828569.311731:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828569/real 1787828569] req@ffff976889cfbc00 x1874670564289408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828569.311746:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564289408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828569.311755:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828569.311759:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828569.311763:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564289408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828569.311786:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564289536:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828569.311806:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828569.311814:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828569/real 1787828569] req@ffff976889cfa580 x1874670564289536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828569.311845:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564289536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828569.311858:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828569.311861:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828569.311864:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564289536:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828569.311869:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564289664:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828569.311886:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828569.311900:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828569/real 1787828569] req@ffff976889cf8000 x1874670564289664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828569.311915:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564289664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828569.311925:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828569.311929:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828569.311934:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564289664:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828569.311957:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564289792:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828569.311977:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828569.311987:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828569/real 1787828569] req@ffff976889cfb0c0 x1874670564289792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828569.311997:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564289792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828569.312004:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828569.312008:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828569.312013:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564289792:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828569.566970:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828569.567073:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828569.567080:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828570.334930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828570.590967:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828570.590993:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828570.591002:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828571.358933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828571.614937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828571.614937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828571.615120:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828572.382896:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828572.638924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828572.638993:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828572.638995:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828573.406976:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787828573.662950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828573.662981:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828573.663095:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828574.430938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828574.431020:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828574.431035:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828574.431042:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828574.431058:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3514 00000100:00080000:2.0:1787828574.431077:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828574.431089:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828574.431152:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564289920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828574.431191:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828574.431197:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828574.431200:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828574.431206:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3514 00000100:00080000:2.0:1787828574.431212:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828574.431219:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828574.431234:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828574.431273:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564289920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828574.431284:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564290048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828574.431309:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828574.431315:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828574.431319:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828574.431325:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3514 00000100:00080000:2.0:1787828574.431331:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828574.431338:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828574.431348:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828574.431369:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564290176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828574.431388:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828574.431394:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828574.431397:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828574.431406:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564290048:192.168.203.139@tcp:38:kworker.0 00000100:00080000:2.0:1787828574.431408:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3514 00000100:00080000:2.0:1787828574.431421:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828574.431430:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828574.431442:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828574.431449:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564290176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828574.431472:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564290304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828574.431494:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828574.431500:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828574.431504:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828574.431512:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3514 00000100:00080000:2.0:1787828574.431520:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828574.431528:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564290304:192.168.203.139@tcp:8:kworker.0 00000100:00080000:2.0:1787828574.431530:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828574.431548:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828574.431576:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564290432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828574.431625:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564290432:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828574.686991:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828574.687068:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828574.687093:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828575.454942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828575.710962:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828575.710971:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828575.710987:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828576.478908:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828576.734917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828576.734917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828576.734984:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828577.502923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:0.0:1787828577.503199:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:0.0:1787828577.503297:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828577.503319:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828577.503401:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828577.503420:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828577.503437:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828577.503453:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:1.0:1787828577.503565:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828574/real 1787828577] req@ffff976889cfbc00 x1874670564290432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828577.503611:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564290432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828577.503635:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828577.503644:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828577.503653:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564290432:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828577.503668:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828574/real 1787828577] req@ffff976889cfa580 x1874670564290304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828577.503692:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564290304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828577.503708:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828577.503713:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828577.503719:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564290304:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828577.503729:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828574/real 1787828577] req@ffff976889cfb0c0 x1874670564290048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828577.503750:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564290048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828577.503765:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828577.503771:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828577.503777:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564290048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828577.503787:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828574/real 1787828577] req@ffff976889cf8000 x1874670564290176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828577.503808:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564290176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828577.503821:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828577.503870:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828577.503877:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564290176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828577.503889:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828574/real 1787828577] req@ffff976889cf8f00 x1874670564289920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828577.503912:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564289920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828577.503925:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828577.503930:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828577.503935:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564289920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828577.758911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828577.759028:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828577.759088:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828578.526933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828578.783021:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828578.783021:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828578.783042:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828579.550919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828579.550970:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828579.550979:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828579.550984:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828579.550995:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3519 00000100:00080000:1.0:1787828579.551025:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828579.551035:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828579.551081:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564290560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828579.551123:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828579.551128:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828579.551131:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828579.551136:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3519 00000100:00080000:1.0:1787828579.551143:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828579.551151:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828579.551182:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828579.551206:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564290688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828579.551235:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564290560:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787828579.551249:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828579.551257:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828579.551260:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828579.551267:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3519 00000100:00080000:1.0:1787828579.551275:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828579.551281:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828579.551290:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828579.551311:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564290816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828579.551326:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828579.551331:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828579.551355:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828579.551356:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787828579.551371:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3519 00000100:00080000:1.0:1787828579.551380:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828579.551387:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828579.551395:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828579.551432:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564290944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828579.551450:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828579.551450:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828579/real 1787828579] req@ffff976889efc780 x1874670564290560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828579.551464:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828579.551470:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828579.551486:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3519 00000100:00080000:1.0:1787828579.551497:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828579.551504:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828579.551511:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564290560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828579.551516:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828579.551559:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828579.551568:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564291072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828579.551570:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828579.551628:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564290560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828579.551678:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564290688:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828579.551723:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828579.551748:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828579/real 1787828579] req@ffff976889eff480 x1874670564290688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828579.551785:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564290688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828579.551801:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828579.551807:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828579.551812:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564290688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828579.551859:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564290816:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828579.551901:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828579.551923:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828579/real 1787828579] req@ffff976889efe580 x1874670564290816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828579.551943:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564290816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828579.551958:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828579.551964:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828579.551969:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564290816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828579.551980:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564290944:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828579.552017:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828579.552037:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828579/real 1787828579] req@ffff976889efe1c0 x1874670564290944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828579.552058:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564290944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828579.552084:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828579.552089:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828579.552095:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564290944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828579.552106:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564291072:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828579.552155:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828579.552175:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828579/real 1787828579] req@ffff976889efcb40 x1874670564291072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828579.552196:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564291072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828579.552239:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828579.552246:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828579.552253:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564291072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828579.806942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828579.806961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828579.807100:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828580.574985:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828580.830927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828580.831135:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828580.831172:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828581.598906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828581.854901:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828581.854911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828581.855070:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828582.622941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828582.878948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828582.878965:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828582.878970:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828583.646971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828583.902985:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828583.902989:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828583.903084:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828584.670951:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828584.670954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828584.670975:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828584.670982:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828584.670992:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3524 00000100:00080000:1.0:1787828584.671004:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828584.671013:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828584.671079:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564291200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828584.671108:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828584.671111:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828584.671112:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828584.671116:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3524 00000100:00080000:1.0:1787828584.671119:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828584.671122:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828584.671144:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828584.671153:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564291328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828584.671161:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828584.671164:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828584.671166:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828584.671169:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3524 00000100:00080000:1.0:1787828584.671171:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828584.671174:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828584.671194:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828584.671205:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564291456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828584.671212:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828584.671215:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828584.671216:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828584.671219:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3524 00000100:00080000:1.0:1787828584.671232:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828584.671235:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828584.671239:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828584.671247:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564291584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828584.671255:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828584.671261:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828584.671263:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564291200:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787828584.671264:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828584.671280:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3524 00000100:00080000:1.0:1787828584.671284:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828584.671287:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828584.671290:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828584.671301:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564291712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828584.671356:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828584.671429:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828584/real 1787828584] req@ffff976889efe940 x1874670564291200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828584.671448:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564291200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828584.671461:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828584.671465:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828584.671469:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564291200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828584.671477:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564291328:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828584.671501:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828584.671513:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828584/real 1787828584] req@ffff976889eff0c0 x1874670564291328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828584.671525:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564291328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828584.671532:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828584.671535:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828584.671538:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564291328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828584.671544:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564291456:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828584.671559:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828584.671567:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828584/real 1787828584] req@ffff976889effc00 x1874670564291456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828584.671579:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564291456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828584.671591:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828584.671596:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828584.671600:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564291456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828584.671653:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564291584:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828584.671695:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828584.671705:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828584/real 1787828584] req@ffff976889efed00 x1874670564291584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828584.671714:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564291584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828584.671720:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828584.671722:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828584.671725:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564291584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828584.671729:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564291712:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828584.671743:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828584.671768:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828584/real 1787828584] req@ffff976889efc3c0 x1874670564291712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828584.671777:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564291712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828584.671783:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828584.671785:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828584.671787:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564291712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828584.926989:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828584.926992:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828584.927032:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828585.694988:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828585.950948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828585.950950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828585.951013:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828586.718991:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828586.974936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828586.974962:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828586.974974:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828587.742907:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828587.998903:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828587.998906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828587.999005:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828588.766904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828589.022913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828589.022914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828589.022939:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828589.790923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828589.790950:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828589.790957:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828589.790962:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828589.790972:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3529 00000100:00080000:2.0:1787828589.790982:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828589.791000:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828589.791040:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564291840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828589.791074:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828589.791077:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828589.791079:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828589.791083:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3529 00000100:00080000:2.0:1787828589.791087:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828589.791091:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828589.791100:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828589.791113:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564291968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828589.791125:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828589.791129:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828589.791130:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828589.791135:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3529 00000100:00080000:2.0:1787828589.791139:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828589.791168:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828589.791187:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828589.791191:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564291840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828589.791201:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564292096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828589.791217:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828589.791220:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828589.791222:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828589.791229:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3529 00000100:00080000:2.0:1787828589.791234:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828589.791241:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828589.791250:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828589.791275:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564292224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828589.791290:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828589.791295:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828589.791299:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828589.791316:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3529 00000100:00080000:2.0:1787828589.791321:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828589.791325:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828589.791330:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828589.791332:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828589.791351:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564292352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828589.791396:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828589/real 1787828589] req@ffff976889cfa940 x1874670564291840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828589.791413:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564291840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828589.791424:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828589.791427:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828589.791431:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564291840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828589.791457:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564291968:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828589.791476:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828589.791485:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828589/real 1787828589] req@ffff976889cfad00 x1874670564291968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828589.791494:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564291968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828589.791505:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828589.791508:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828589.791511:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564291968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828589.791515:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564292096:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828589.791529:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828589.791534:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828589/real 1787828589] req@ffff976889cf92c0 x1874670564292096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828589.791542:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564292096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828589.791550:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828589.791554:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828589.791557:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564292096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828589.791579:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564292224:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828589.791604:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828589.791618:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828589/real 1787828589] req@ffff976889cf9e00 x1874670564292224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828589.791647:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564292224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828589.791656:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828589.791660:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828589.791664:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564292224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828589.791670:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564292352:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828589.791692:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828589.791702:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828589/real 1787828589] req@ffff976889cfb480 x1874670564292352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828589.791728:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564292352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828589.791733:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828589.791735:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828589.791737:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564292352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828590.046886:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828590.046891:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828590.046993:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828590.814923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828591.071004:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828591.071101:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828591.071193:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828591.838897:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828592.094907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828592.094907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828592.094961:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828592.862920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828593.118934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828593.118962:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828593.118966:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828593.886929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828594.142953:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828594.142957:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828594.143000:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828594.910925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828594.910977:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828594.910982:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828594.910986:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828594.910995:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3534 00000100:00080000:3.0:1787828594.911006:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828594.911014:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828594.911050:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564292480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828594.911079:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828594.911083:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828594.911085:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828594.911089:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3534 00000100:00080000:3.0:1787828594.911093:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828594.911097:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828594.911121:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828594.911133:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564292480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828594.911138:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564292608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828594.911152:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828594.911155:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828594.911157:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828594.911162:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3534 00000100:00080000:3.0:1787828594.911167:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828594.911171:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828594.911176:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828594.911189:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564292736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828594.911199:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828594.911202:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828594.911204:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828594.911207:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787828594.911211:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3534 00000100:00080000:3.0:1787828594.911216:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828594.911219:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828594.911224:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828594.911244:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828594/real 1787828594] req@ffff97688cb5f0c0 x1874670564292480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828594.911263:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564292480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828594.911275:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828594.911279:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828594.911283:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564292480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828594.911310:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564292864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828594.911311:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564292608:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787828594.911326:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828594.911329:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828594.911331:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828594.911335:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3534 00000800:00000100:2.0:1787828594.911339:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787828594.911340:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828594.911347:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828594.911351:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828594.911355:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828594/real 1787828594] req@ffff97688cb5d2c0 x1874670564292608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828594.911365:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564292992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828594.911369:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564292608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828594.911395:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828594.911400:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828594.911403:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564292608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828594.911407:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564292736:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828594.911422:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828594.911430:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828594/real 1787828594] req@ffff97688cb5e1c0 x1874670564292736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828594.911437:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564292736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828594.911443:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828594.911445:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828594.911447:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564292736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828594.911461:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564292864:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828594.911473:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828594.911480:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828594/real 1787828594] req@ffff97688cb5c780 x1874670564292864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828594.911488:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564292864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828594.911498:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828594.911502:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828594.911506:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564292864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828594.911514:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564292992:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828594.911535:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828594.911550:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828594/real 1787828594] req@ffff97688cb5c3c0 x1874670564292992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828594.911569:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564292992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828594.911579:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828594.911582:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828594.911586:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564292992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828595.166940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828595.166941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828595.166965:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828595.934934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828596.190922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828596.190938:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828596.191033:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828596.958936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828597.214985:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828597.214986:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828597.215030:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828597.982942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828598.238931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828598.238936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828598.239009:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828599.006910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828599.262928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828599.262933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828599.262971:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828600.030976:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828600.030981:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828600.030992:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828600.030996:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828600.031005:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3539 00000100:00080000:2.0:1787828600.031016:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828600.031021:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828600.031055:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564293120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828600.031108:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828600.031113:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828600.031115:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828600.031120:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3539 00000100:00080000:2.0:1787828600.031124:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828600.031130:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828600.031142:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828600.031159:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564293248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828600.031169:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564293120:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828600.031182:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828600.031186:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828600.031187:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828600.031191:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3539 00000100:00080000:2.0:1787828600.031195:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828600.031199:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828600.031204:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828600.031214:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564293376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828600.031223:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828600.031228:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828600.031231:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828600.031236:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3539 00000100:00080000:2.0:1787828600.031241:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828600.031246:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828600.031252:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1787828600.031258:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828600.031269:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564293504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828600.031282:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828600.031285:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828600.031287:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828600.031292:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3539 00000100:00080000:2.0:1787828600.031295:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828600.031301:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828600.031307:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828600.031311:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828600/real 1787828600] req@ffff976889cfa1c0 x1874670564293120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828600.031321:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564293632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828600.031333:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564293120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828600.031345:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828600.031348:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828600.031353:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564293120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828600.031378:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564293248:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828600.031395:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828600.031404:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828600/real 1787828600] req@ffff976889cfb480 x1874670564293248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828600.031412:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564293248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828600.031418:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828600.031421:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828600.031423:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564293248:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828600.031427:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564293376:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828600.031437:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828600.031443:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828600/real 1787828600] req@ffff976889cf9e00 x1874670564293376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828600.031450:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564293376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828600.031456:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828600.031458:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828600.031459:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564293376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828600.031463:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564293504:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828600.031474:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828600.031481:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828600/real 1787828600] req@ffff976889cf92c0 x1874670564293504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828600.031487:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564293504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828600.031493:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828600.031495:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828600.031496:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564293504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828600.031500:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564293632:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828600.031510:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828600.031515:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828600/real 1787828600] req@ffff976889cfad00 x1874670564293632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828600.031522:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564293632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828600.031528:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828600.031530:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828600.031531:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564293632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828600.286924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828600.286987:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828600.287018:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828601.054904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828601.310933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828601.310940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828601.310951:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828602.078977:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828602.334976:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828602.334984:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828602.335079:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828603.102890:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828603.358901:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828603.358905:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828603.358920:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828604.126960:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828604.382936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828604.382936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828604.382996:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828605.150918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828605.150935:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828605.150940:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828605.150944:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828605.150952:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3545 00000100:00080000:0.0:1787828605.150961:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828605.150967:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828605.151001:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564293760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828605.151023:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828605.151026:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828605.151027:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828605.151030:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3545 00000100:00080000:0.0:1787828605.151046:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828605.151051:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828605.151066:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828605.151068:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564293760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828605.151084:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564293888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828605.151094:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828605.151097:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828605.151098:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828605.151101:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3545 00000100:00080000:0.0:1787828605.151104:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828605.151106:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828605.151110:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828605.151117:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564294016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828605.151124:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828605.151127:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828605.151128:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828605.151133:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3545 00000800:00000100:1.0:1787828605.151134:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828605.151136:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828605.151140:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828605.151143:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828605.151151:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564294144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828605.151162:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828605.151164:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828605.151166:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828605.151168:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3545 00000100:00080000:0.0:1787828605.151171:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828605.151175:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828605.151179:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828605.151180:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828605/real 1787828605] req@ffff976882aed680 x1874670564293760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828605.151195:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564294272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828605.151203:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564293760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828605.151219:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828605.151223:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828605.151228:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564293760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828605.151251:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564293888:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828605.151273:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828605.151286:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828605/real 1787828605] req@ffff976882aee940 x1874670564293888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828605.151298:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564293888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828605.151306:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828605.151309:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828605.151312:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564293888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828605.151318:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564294016:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828605.151334:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828605.151343:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828605/real 1787828605] req@ffff976882aef0c0 x1874670564294016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828605.151354:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564294016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828605.151362:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828605.151364:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828605.151367:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564294016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828605.151372:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564294144:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828605.151397:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828605.151407:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828605/real 1787828605] req@ffff976882aeed00 x1874670564294144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828605.151420:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564294144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828605.151428:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828605.151431:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828605.151434:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564294144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828605.151440:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564294272:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828605.151457:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828605.151468:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828605/real 1787828605] req@ffff976882aee580 x1874670564294272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828605.151479:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564294272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828605.151493:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828605.151496:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828605.151499:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564294272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828605.406915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828605.406922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828605.406964:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828606.174899:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828606.430908:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828606.430928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828606.430957:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828607.198917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828607.454893:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828607.454896:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828607.454930:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828608.222956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828608.478894:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828608.478894:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828608.478958:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828609.246921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828609.502887:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828609.502888:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828609.502925:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828610.270909:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828610.270909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828610.270918:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828610.270923:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828610.270931:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3550 00000100:00080000:0.0:1787828610.270939:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828610.270945:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828610.270978:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564294400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828610.271011:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828610.271016:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828610.271018:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828610.271023:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3550 00000100:00080000:0.0:1787828610.271027:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828610.271030:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828610.271038:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564294400:192.168.203.139@tcp:250:kworker.0 00000008:00100000:0.0:1787828610.271039:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828610.271056:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564294528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828610.271065:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828610.271068:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828610.271069:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828610.271072:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3550 00000100:00080000:0.0:1787828610.271088:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828610.271092:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:1.0:1787828610.271097:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:0.0:1787828610.271101:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828610.271119:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564294656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828610.271131:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828610.271132:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828610/real 1787828610] req@ffff976882aec3c0 x1874670564294400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828610.271138:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828610.271142:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828610.271151:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3550 00000100:00080000:0.0:1787828610.271157:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828610.271161:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828610.271166:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828610.271169:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564294400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828610.271178:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564294784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828610.271189:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828610.271199:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:0.0:1787828610.271201:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828610.271205:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564294400:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828610.271206:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828610.271211:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828610.271216:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3550 00000100:00080000:0.0:1787828610.271219:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828610.271221:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828610.271224:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828610.271231:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564294912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828610.271235:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564294528:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828610.271264:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828610.271276:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828610/real 1787828610] req@ffff976882aee580 x1874670564294528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828610.271284:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564294528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828610.271291:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828610.271293:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828610.271295:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564294528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828610.271299:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564294656:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828610.271309:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828610.271315:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828610/real 1787828610] req@ffff976882aeed00 x1874670564294656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828610.271322:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564294656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828610.271328:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828610.271329:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828610.271331:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564294656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828610.271335:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564294784:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828610.271345:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828610.271351:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828610/real 1787828610] req@ffff976882aef0c0 x1874670564294784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828610.271358:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564294784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828610.271364:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828610.271366:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828610.271368:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564294784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828610.271378:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564294912:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828610.271388:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828610.271393:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828610/real 1787828610] req@ffff976882aee940 x1874670564294912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828610.271400:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564294912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828610.271405:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828610.271407:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828610.271409:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564294912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828610.526912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828610.526912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828610.526980:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828611.294907:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828611.550918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828611.550934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828611.550952:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828612.318954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828612.574932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828612.574932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828612.575001:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828613.342894:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828613.598938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828613.598938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828613.599008:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828614.366950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828614.622930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828614.622931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828614.623011:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828615.390911:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828615.390915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828615.390925:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828615.390931:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828615.390941:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3555 00000100:00080000:1.0:1787828615.390951:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828615.390959:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828615.391022:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564295040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828615.391056:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828615.391060:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828615.391063:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828615.391067:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3555 00000100:00080000:1.0:1787828615.391071:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828615.391077:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828615.391087:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828615.391094:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564295040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828615.391107:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564295168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828615.391121:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828615.391125:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828615.391128:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828615.391132:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3555 00000100:00080000:1.0:1787828615.391137:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828615.391142:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828615.391149:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828615.391152:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828615.391167:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564295296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828615.391192:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828615/real 1787828615] req@ffff976889efc3c0 x1874670564295040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828615.391198:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828615.391207:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828615.391212:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828615.391222:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3555 00000100:00080000:1.0:1787828615.391229:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828615.391234:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828615.391237:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564295040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828615.391243:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00080000:2.0:1787828615.391248:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828615.391252:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828615.391256:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564295040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828615.391263:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564295424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828615.391275:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828615.391280:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564295168:192.168.203.139@tcp:38:kworker.0 00000100:00080000:1.0:1787828615.391282:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828615.391287:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828615.391293:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3555 00000100:00080000:1.0:1787828615.391298:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828615.391303:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:2.0:1787828615.391306:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787828615.391312:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828615.391320:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828615/real 1787828615] req@ffff976889efed00 x1874670564295168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828615.391330:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564295552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828615.391333:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564295168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828615.391360:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828615.391363:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828615.391366:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564295168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828615.391369:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564295296:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828615.391386:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828615.391394:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828615/real 1787828615] req@ffff976889effc00 x1874670564295296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828615.391402:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564295296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828615.391412:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828615.391414:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828615.391417:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564295296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828615.391420:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564295424:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828615.391430:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828615.391436:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828615/real 1787828615] req@ffff976889eff0c0 x1874670564295424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828615.391443:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564295424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828615.391448:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828615.391450:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828615.391452:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564295424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828615.391462:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564295552:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828615.391472:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828615.391478:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828615/real 1787828615] req@ffff976889efe940 x1874670564295552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828615.391485:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564295552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828615.391490:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828615.391492:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828615.391494:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564295552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828615.646895:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828615.646896:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828615.647069:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828616.414926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828616.670943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828616.670966:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828616.671028:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828617.438975:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828617.694913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828617.694913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828617.695004:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828618.462907:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828618.718932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828618.718932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828618.718984:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828619.486898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828619.742891:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828619.742902:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828619.742952:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828620.510900:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828620.510909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828620.510910:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828620.510914:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828620.510924:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3560 00000100:00080000:1.0:1787828620.510943:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828620.510949:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828620.510982:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564295680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828620.511004:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828620.511007:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828620.511008:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828620.511011:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3560 00000100:00080000:1.0:1787828620.511014:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828620.511016:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828620.511024:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828620.511032:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564295808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828620.511041:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828620.511044:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828620.511044:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564295680:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787828620.511048:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828620.511055:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3560 00000100:00080000:1.0:1787828620.511057:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828620.511059:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828620.511062:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828620.511072:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564295936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828620.511078:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828620.511080:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828620.511081:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828620.511085:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3560 00000100:00080000:1.0:1787828620.511088:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828620.511090:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828620.511093:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828620.511101:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564296064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828620.511107:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828620.511109:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828620.511111:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828620.511115:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787828620.511118:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3560 00000100:00080000:1.0:1787828620.511122:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828620.511125:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828620.511128:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828620.511135:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564296192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828620.511165:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828620/real 1787828620] req@ffff976889efcb40 x1874670564295680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828620.511183:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564295680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828620.511193:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828620.511197:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828620.511201:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564295680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828620.511223:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564295808:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828620.511246:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828620.511257:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828620/real 1787828620] req@ffff976889efe1c0 x1874670564295808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828620.511266:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564295808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828620.511272:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828620.511274:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828620.511277:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564295808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828620.511281:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564295936:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828620.511293:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828620.511298:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828620/real 1787828620] req@ffff976889efe580 x1874670564295936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828620.511306:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564295936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828620.511311:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828620.511313:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828620.511315:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564295936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828620.511318:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564296064:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828620.511328:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828620.511334:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828620/real 1787828620] req@ffff976889eff480 x1874670564296064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828620.511341:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564296064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828620.511346:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828620.511348:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828620.511350:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564296064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828620.511353:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564296192:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828620.511363:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828620.511369:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828620/real 1787828620] req@ffff976889efc780 x1874670564296192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828620.511376:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564296192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828620.511381:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828620.511383:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828620.511384:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564296192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828620.766899:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828620.766899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828620.766941:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828621.534912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828621.790909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828621.790953:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828621.790973:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828622.558902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828622.814920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828622.814945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828622.814958:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828623.582942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828623.838871:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828623.838882:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828623.838962:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828624.606923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828624.862875:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828624.862940:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828624.862955:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828625.630952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828625.630986:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828625.630994:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828625.630999:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828625.631012:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3565 00000100:00080000:2.0:1787828625.631026:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828625.631034:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828625.631081:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564296320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828625.631107:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828625.631110:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828625.631111:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828625.631114:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3565 00000100:00080000:2.0:1787828625.631118:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828625.631121:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828625.631132:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828625.631141:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564296448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828625.631149:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828625.631153:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828625.631156:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828625.631157:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564296320:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828625.631161:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3565 00000100:00080000:2.0:1787828625.631166:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828625.631170:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828625.631172:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828625.631179:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564296576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828625.631186:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828625.631188:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828625.631190:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828625.631192:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3565 00000100:00080000:2.0:1787828625.631194:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828625.631196:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828625.631200:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828625.631208:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564296704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828625.631214:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828625.631217:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828625.631218:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828625.631221:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3565 00000100:00080000:2.0:1787828625.631224:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828625.631226:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828625.631228:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828625.631235:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564296832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828625.631250:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828625.631312:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828625/real 1787828625] req@ffff976889cfa940 x1874670564296320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828625.631335:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564296320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828625.631348:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828625.631353:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828625.631357:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564296320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828625.631384:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564296448:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828625.631411:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828625.631426:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828625/real 1787828625] req@ffff976889cf8f00 x1874670564296448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828625.631440:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564296448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828625.631450:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828625.631454:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828625.631458:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564296448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828625.631464:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564296576:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828625.631489:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828625.631501:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828625/real 1787828625] req@ffff976889cf8000 x1874670564296576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828625.631515:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564296576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828625.631523:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828625.631527:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828625.631530:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564296576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828625.631536:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564296704:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828625.631559:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828625.631573:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828625/real 1787828625] req@ffff976889cfb0c0 x1874670564296704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828625.631585:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564296704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828625.631594:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828625.631597:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828625.631601:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564296704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828625.631607:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564296832:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828625.631628:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828625.631639:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828625/real 1787828625] req@ffff976889cfa580 x1874670564296832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828625.631657:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564296832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828625.631666:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828625.631669:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828625.631672:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564296832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828625.886904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828625.886914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828625.886968:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828626.654900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828626.910915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828626.910915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828626.910939:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828627.678914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828627.934917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828627.934939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828627.934994:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828628.702923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828628.958893:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828628.958914:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828628.958966:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828629.726919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828629.982937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828629.982971:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828629.982995:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828630.750926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828630.750988:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828630.751001:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828630.751006:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828630.751019:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3570 00000100:00080000:0.0:1787828630.751031:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828630.751038:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828630.751083:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564296960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828630.751109:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828630.751112:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828630.751114:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828630.751118:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3570 00000100:00080000:0.0:1787828630.751121:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828630.751125:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828630.751138:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828630.751156:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564297088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828630.751171:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564296960:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828630.751175:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828630.751187:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828630.751191:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828630.751198:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3570 00000100:00080000:0.0:1787828630.751204:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828630.751208:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828630.751214:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828630.751226:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564297216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828630.751235:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828630.751238:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828630.751239:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828630.751243:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3570 00000100:00080000:0.0:1787828630.751247:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828630.751250:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828630.751254:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1787828630.751261:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828630.751263:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564297344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828630.751272:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828630.751275:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828630.751277:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828630.751283:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3570 00000100:00080000:0.0:1787828630.751291:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828630.751295:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828630.751299:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828630.751311:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564297472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828630.751321:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828630/real 1787828630] req@ffff976882aed680 x1874670564296960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828630.751346:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564296960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828630.751359:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828630.751365:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828630.751369:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564296960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828630.751395:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564297088:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828630.751424:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828630.751441:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828630/real 1787828630] req@ffff976882aee940 x1874670564297088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828630.751457:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564297088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828630.751468:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828630.751474:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828630.751478:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564297088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828630.751487:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564297216:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828630.751517:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828630.751532:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828630/real 1787828630] req@ffff976882aef0c0 x1874670564297216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828630.751552:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564297216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828630.751567:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828630.751572:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828630.751577:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564297216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828630.751584:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564297344:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828630.751606:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828630.751618:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828630/real 1787828630] req@ffff976882aeed00 x1874670564297344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828630.751630:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564297344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828630.751645:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828630.751649:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828630.751652:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564297344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828630.751659:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564297472:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828630.751681:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828630.751693:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828630/real 1787828630] req@ffff976882aee580 x1874670564297472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828630.751705:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564297472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828630.751714:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828630.751717:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828630.751720:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564297472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828631.006948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828631.006958:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828631.007000:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828631.774952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828632.030970:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828632.030976:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828632.031038:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828632.798917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828633.054965:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828633.055003:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828633.055003:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828633.822942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828634.078944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828634.078956:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828634.078980:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828634.846888:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828635.102969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828635.103013:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828635.103035:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828635.870975:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828635.871006:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828635.871018:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828635.871024:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828635.871036:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3575 00000100:00080000:2.0:1787828635.871051:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828635.871076:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828635.871144:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564297600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828635.871198:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828635.871202:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828635.871204:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828635.871209:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3575 00000100:00080000:2.0:1787828635.871213:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828635.871216:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828635.871252:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828635.871273:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564297728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828635.871288:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828635.871291:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564297600:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828635.871294:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828635.871299:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828635.871318:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3575 00000100:00080000:2.0:1787828635.871323:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828635.871329:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828635.871347:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828635.871386:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564297856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828635.871407:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828635.871413:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828635.871417:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828635.871423:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3575 00000100:00080000:2.0:1787828635.871447:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828635.871465:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828635.871472:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1787828635.871476:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828635.871495:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564297984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828635.871510:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828635.871515:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828635.871529:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828635.871535:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3575 00000100:00080000:2.0:1787828635.871540:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828635.871545:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828635.871551:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828635/real 1787828635] req@ffff976889cfbc00 x1874670564297600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828635.871554:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828635.871573:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564298112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828635.871605:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564297600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828635.871625:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828635.871631:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828635.871635:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564297600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828635.871663:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564297728:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828635.871693:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828635.871712:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828635/real 1787828635] req@ffff976889cfa580 x1874670564297728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828635.871731:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564297728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828635.871754:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828635.871757:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828635.871760:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564297728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828635.871767:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564297856:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828635.871788:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828635.871807:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828635/real 1787828635] req@ffff976889cfb0c0 x1874670564297856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828635.871886:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564297856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828635.871898:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828635.871909:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828635.871912:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564297856:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828635.871919:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564297984:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828635.871942:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828635.871952:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828635/real 1787828635] req@ffff976889cf8000 x1874670564297984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828635.871962:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564297984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828635.871970:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828635.871972:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828635.871975:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564297984:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828635.871980:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564298112:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828635.871993:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828635.872000:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828635/real 1787828635] req@ffff976889cf8f00 x1874670564298112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828635.872019:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564298112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828635.872029:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828635.872033:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828635.872037:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564298112:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828636.126973:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828636.127006:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828636.127057:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828636.894900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828637.151043:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828637.151064:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828637.151195:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828637.918934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828638.174941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828638.174941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828638.174987:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828638.942970:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828639.198904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828639.198904:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828639.198929:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828639.966925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828640.222921:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828640.222922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828640.223071:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828640.990980:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828640.990986:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828640.990992:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828640.991000:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828640.991011:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3580 00000100:00080000:3.0:1787828640.991023:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828640.991031:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828640.991072:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564298240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828640.991098:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828640.991100:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828640.991102:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828640.991105:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3580 00000100:00080000:3.0:1787828640.991109:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828640.991112:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828640.991120:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828640.991130:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564298368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828640.991140:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828640.991144:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828640.991146:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828640.991152:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3580 00000100:00100000:1.0:1787828640.991153:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564298240:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787828640.991157:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828640.991165:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828640.991170:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828640.991186:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564298496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828640.991195:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828640.991198:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828640.991199:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828640.991202:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3580 00000100:00080000:3.0:1787828640.991205:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828640.991209:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828640.991213:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828640.991221:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564298624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828640.991228:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828640.991231:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828640.991233:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828640.991237:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3580 00000100:00080000:3.0:1787828640.991242:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828640.991248:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828640.991254:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828640.991262:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564298368:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828640.991270:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564298752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828640.991319:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564298496:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828640.991351:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564298624:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828640.991374:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564298752:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828641.246923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828641.246945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828641.246950:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828642.015062:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828642.270951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828642.270953:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828642.270968:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828643.039005:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828643.294938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828643.294938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828643.294964:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828644.063127:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:1.0:1787828644.063155:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:1.0:1787828644.063200:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787828644.063222:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787828644.063276:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787828644.063290:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:2.0:1787828644.063302:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828640/real 1787828644] req@ffff97688cb5c3c0 x1874670564298240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787828644.063303:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787828644.063317:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:2.0:1787828644.063340:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564298240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828644.063356:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828644.063363:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828644.063368:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564298240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828644.063377:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828640/real 1787828644] req@ffff97688cb5c780 x1874670564298368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828644.063392:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564298368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828644.063402:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828644.063407:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828644.063411:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564298368:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828644.063447:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828640/real 1787828644] req@ffff97688cb5e1c0 x1874670564298496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828644.063461:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564298496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828644.063470:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828644.063474:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828644.063477:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564298496:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828644.063484:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828640/real 1787828644] req@ffff97688cb5d2c0 x1874670564298624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828644.063497:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564298624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828644.063506:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828644.063509:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828644.063512:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564298624:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828644.063519:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828640/real 1787828644] req@ffff97688cb5f0c0 x1874670564298752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828644.063531:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564298752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828644.063538:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828644.063542:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828644.063545:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564298752:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828644.318907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828644.318907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828644.318929:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828645.086996:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828645.342926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828645.342969:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828645.342977:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00080000:3.0:1787828646.110928:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828646.110941:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828646.110947:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828646.110959:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3586 00000100:00080000:3.0:1787828646.110985:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828646.110994:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828646.111026:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787828646.111041:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564298880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828646.111064:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828646.111068:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828646.111071:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828646.111075:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3586 00000100:00080000:3.0:1787828646.111079:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828646.111082:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828646.111093:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828646.111110:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564299008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828646.111111:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564298880:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787828646.111143:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828646.111146:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828646.111147:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828646.111150:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3586 00000100:00080000:3.0:1787828646.111152:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828646.111155:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828646.111158:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828646.111167:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564299136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828646.111174:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828646.111177:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828646.111178:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828646.111182:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3586 00000100:00080000:3.0:1787828646.111207:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828646.111210:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828646.111216:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1787828646.111229:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828646.111232:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564299264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828646.111248:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828646.111254:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828646.111257:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828646.111262:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3586 00000100:00080000:3.0:1787828646.111268:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828646.111273:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828646.111279:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828646.111305:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564299392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828646.111306:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828646/real 1787828646] req@ffff97688cb5d680 x1874670564298880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828646.111369:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564298880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828646.111391:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828646.111397:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828646.111402:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564298880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828646.111431:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564299008:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828646.111467:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828646.111483:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828646/real 1787828646] req@ffff97688cb5c000 x1874670564299008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828646.111497:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564299008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828646.111506:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828646.111510:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828646.111513:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564299008:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828646.111521:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564299136:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828646.111557:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828646.111570:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828646/real 1787828646] req@ffff97688cb5cb40 x1874670564299136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828646.111583:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564299136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828646.111592:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828646.111596:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828646.111600:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564299136:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828646.111607:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564299264:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828646.111630:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828646.111642:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828646/real 1787828646] req@ffff97688cb5de00 x1874670564299264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828646.111655:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564299264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828646.111664:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828646.111668:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828646.111672:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564299264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828646.111679:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564299392:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828646.111699:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828646.111710:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828646/real 1787828646] req@ffff97688cb5f840 x1874670564299392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828646.111722:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564299392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828646.111731:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828646.111735:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828646.111739:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564299392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828646.366916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828646.366916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828646.366938:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828647.135111:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828647.390978:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828647.390981:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828647.391006:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828648.158956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828648.414903:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828648.414942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828648.414967:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828649.182944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828649.438995:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828649.438996:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828649.439087:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828650.206930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828650.462917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828650.462919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828650.462948:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828651.230936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828651.230953:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828651.230963:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828651.230968:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828651.230982:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3591 00000100:00080000:3.0:1787828651.230998:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828651.231010:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828651.231067:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564299520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828651.231104:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828651.231109:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828651.231112:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828651.231119:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3591 00000100:00080000:3.0:1787828651.231124:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828651.231128:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828651.231140:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828651.231144:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564299520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828651.231160:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564299648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828651.231172:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828651.231176:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828651.231178:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828651.231199:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3591 00000100:00080000:3.0:1787828651.231214:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828651.231219:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:2.0:1787828651.231224:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:3.0:1787828651.231232:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828651.231254:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564299776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828651.231265:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828651/real 1787828651] req@ffff97688cb5f840 x1874670564299520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828651.231274:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828651.231285:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828651.231287:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828651.231298:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3591 00000100:00100000:2.0:1787828651.231300:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564299520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828651.231302:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828651.231310:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828651.231317:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00080000:2.0:1787828651.231326:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828651.231333:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564299904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828651.231341:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:3.0:1787828651.231351:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828651.231355:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828651.231358:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564299520:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787828651.231358:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828651.231363:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3591 00000100:00080000:3.0:1787828651.231366:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828651.231368:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828651.231373:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828651.231390:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564300032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828651.231415:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564299648:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828651.231438:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828651.231449:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828651/real 1787828651] req@ffff97688cb5de00 x1874670564299648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828651.231458:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564299648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828651.231464:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828651.231467:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828651.231469:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564299648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828651.231474:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564299776:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828651.231485:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828651.231491:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828651/real 1787828651] req@ffff97688cb5cb40 x1874670564299776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828651.231499:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564299776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828651.231504:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828651.231506:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828651.231508:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564299776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828651.231512:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564299904:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828651.231522:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828651.231528:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828651/real 1787828651] req@ffff97688cb5c000 x1874670564299904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828651.231535:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564299904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828651.231541:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828651.231542:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828651.231544:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564299904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828651.231548:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564300032:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828651.231557:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828651.231563:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828651/real 1787828651] req@ffff97688cb5d680 x1874670564300032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828651.231571:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564300032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828651.231576:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828651.231577:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828651.231579:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564300032:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828651.486929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828651.486933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828651.486950:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828652.254892:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828652.510903:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828652.510905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828652.511034:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828653.278927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828653.534963:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828653.534969:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828653.535039:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828654.302917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828654.558950:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828654.558965:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828654.558976:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828655.326941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828655.582963:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828655.582975:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828655.583025:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828656.350973:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828656.350977:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828656.350985:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828656.350991:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828656.350999:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3596 00000100:00080000:0.0:1787828656.351008:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828656.351014:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828656.351052:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564300160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828656.351081:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828656.351085:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828656.351087:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828656.351092:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3596 00000100:00080000:0.0:1787828656.351096:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828656.351102:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828656.351113:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828656.351121:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564300160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828656.351131:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564300288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828656.351140:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828656.351143:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828656.351144:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828656.351146:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3596 00000100:00080000:0.0:1787828656.351149:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828656.351152:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828656.351157:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828656.351165:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564300416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828656.351173:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828656.351175:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828656.351177:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828656.351179:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3596 00000100:00080000:0.0:1787828656.351182:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828656.351184:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828656.351187:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1787828656.351191:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828656.351198:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564300544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828656.351205:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828656.351208:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828656.351210:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828656.351213:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3596 00000100:00080000:0.0:1787828656.351216:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828656.351218:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828656.351221:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828656.351228:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564300672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828656.351243:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828656/real 1787828656] req@ffff976882aec3c0 x1874670564300160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828656.351262:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564300160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828656.351275:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828656.351279:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828656.351284:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564300160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828656.351306:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564300288:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828656.351331:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828656.351345:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828656/real 1787828656] req@ffff976882aee580 x1874670564300288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828656.351358:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564300288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828656.351367:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828656.351371:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828656.351374:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564300288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828656.351380:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564300416:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828656.351401:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828656.351413:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828656/real 1787828656] req@ffff976882aeed00 x1874670564300416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828656.351426:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564300416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828656.351435:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828656.351440:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828656.351444:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564300416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828656.351454:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564300544:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828656.351478:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828656.351490:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828656/real 1787828656] req@ffff976882aef0c0 x1874670564300544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828656.351504:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564300544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828656.351512:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828656.351516:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828656.351519:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564300544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828656.351525:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564300672:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828656.351547:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828656.351564:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828656/real 1787828656] req@ffff976882aee940 x1874670564300672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828656.351577:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564300672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828656.351585:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828656.351588:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828656.351592:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564300672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828656.606911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828656.606912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828656.606922:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828657.374897:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828657.630919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828657.630947:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828657.631120:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828658.398968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828658.654904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828658.654938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828658.654949:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828659.423021:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828659.678951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828659.678952:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828659.679006:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828660.446958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828660.702936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828660.702954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828660.703012:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828661.470924:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828661.470933:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828661.470937:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828661.470946:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3601 00000100:00100000:2.0:1787828661.470951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828661.470956:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828661.470965:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828661.471022:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564300800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828661.471066:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828661.471069:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828661.471071:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828661.471074:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3601 00000100:00080000:0.0:1787828661.471077:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828661.471082:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828661.471097:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828661.471112:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564300800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828661.471113:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564300928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828661.471139:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828661.471142:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828661.471144:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828661.471149:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3601 00000100:00080000:0.0:1787828661.471154:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828661.471159:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828661.471164:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828661.471179:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564301056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828661.471190:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828661.471193:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828661.471195:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828661.471198:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3601 00000100:00080000:0.0:1787828661.471210:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828661.471214:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828661.471218:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1787828661.471225:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828661.471232:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564301184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828661.471242:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828661.471245:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828661.471248:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828661.471252:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3601 00000100:00080000:0.0:1787828661.471257:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828661.471260:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828661.471263:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828661.471270:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564301312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828661.471309:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828661/real 1787828661] req@ffff976882aed680 x1874670564300800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828661.471331:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564300800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828661.471346:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828661.471351:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828661.471356:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564300800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828661.471386:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564300928:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828661.471413:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828661.471428:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828661/real 1787828661] req@ffff976882aee940 x1874670564300928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828661.471442:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564300928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828661.471450:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828661.471454:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828661.471458:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564300928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828661.471465:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564301056:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828661.471485:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828661.471497:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828661/real 1787828661] req@ffff976882aef0c0 x1874670564301056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828661.471527:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564301056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828661.471536:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828661.471540:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828661.471543:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564301056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828661.471550:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564301184:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828661.471571:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828661.471591:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828661/real 1787828661] req@ffff976882aeed00 x1874670564301184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828661.471604:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564301184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828661.471613:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828661.471619:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828661.471623:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564301184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828661.471629:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564301312:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828661.471677:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828661.471690:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828661/real 1787828661] req@ffff976882aee580 x1874670564301312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828661.471712:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564301312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828661.471727:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828661.471731:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828661.471734:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564301312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828661.726947:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828661.726958:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828661.727038:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828662.494924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828662.750987:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828662.750988:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828662.751054:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828663.519006:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828663.774937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828663.774943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828663.775033:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828664.543044:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828664.798951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828664.798963:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828664.799021:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828665.566940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828665.822911:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828665.822923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828665.822976:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828666.590953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828666.590969:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828666.590979:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828666.590985:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828666.590999:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3606 00000100:00080000:0.0:1787828666.591015:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828666.591028:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828666.591082:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564301440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828666.591116:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828666.591120:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828666.591122:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828666.591127:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3606 00000100:00080000:0.0:1787828666.591132:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828666.591137:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828666.591150:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828666.591164:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564301440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828666.591175:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564301568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828666.591192:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828666.591196:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828666.591217:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828666.591222:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3606 00000100:00080000:0.0:1787828666.591227:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828666.591231:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828666.591236:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828666.591253:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564301696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828666.591265:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828666.591269:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828666.591272:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828666.591279:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828666.591284:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3606 00000100:00080000:0.0:1787828666.591291:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828666.591298:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828666.591306:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828666.591321:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564301824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828666.591330:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828666.591333:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787828666.591337:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828666/real 1787828666] req@ffff976882aec3c0 x1874670564301440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828666.591338:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828666.591347:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3606 00000100:00080000:0.0:1787828666.591357:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828666.591362:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828666.591382:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828666.591384:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564301440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828666.591395:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564301952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828666.591397:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828666.591407:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828666.591414:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564301440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828666.591438:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564301568:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828666.591459:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828666.591469:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828666/real 1787828666] req@ffff976882aee580 x1874670564301568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828666.591479:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564301568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828666.591490:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828666.591493:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828666.591496:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564301568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828666.591499:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564301696:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828666.591514:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828666.591523:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828666/real 1787828666] req@ffff976882aeed00 x1874670564301696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828666.591532:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564301696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828666.591537:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828666.591539:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828666.591541:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564301696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828666.591546:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564301824:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828666.591560:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828666.591568:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828666/real 1787828666] req@ffff976882aef0c0 x1874670564301824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828666.591576:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564301824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828666.591581:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828666.591583:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828666.591588:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564301824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828666.591592:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564301952:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828666.591622:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828666.591641:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828666/real 1787828666] req@ffff976882aee940 x1874670564301952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828666.591663:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564301952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828666.591672:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828666.591675:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828666.591677:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564301952:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828666.846931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828666.846931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828666.847040:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828667.614937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828667.870975:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828667.870975:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828667.871033:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828668.638964:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828668.894949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828668.894956:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828668.895028:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828669.662932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828669.918932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828669.918938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828669.919000:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828670.686933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828670.943014:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828670.943058:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828670.943082:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828671.710932:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828671.710944:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828671.710949:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828671.710953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828671.710964:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3611 00000100:00080000:1.0:1787828671.710977:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828671.710986:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828671.711027:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564302080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828671.711058:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828671.711062:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828671.711064:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828671.711069:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3611 00000100:00080000:1.0:1787828671.711074:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828671.711079:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828671.711090:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828671.711098:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564302080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828671.711107:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564302208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828671.711122:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828671.711126:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828671.711128:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828671.711134:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3611 00000100:00080000:1.0:1787828671.711138:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828671.711143:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828671.711150:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828671.711164:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564302336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828671.711175:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828671.711179:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828671.711181:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828671.711189:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787828671.711195:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3611 00000100:00080000:1.0:1787828671.711214:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828671.711220:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828671.711226:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828671.711238:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828671/real 1787828671] req@ffff976889efc780 x1874670564302080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828671.711246:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564302464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828671.711316:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828671.711320:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828671.711322:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828671.711329:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564302080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828671.711334:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3611 00000100:00080000:1.0:1787828671.711345:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828671.711356:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:2.0:1787828671.711358:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787828671.711364:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828671.711365:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828671.711384:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564302080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828671.711389:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564302592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828671.711421:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564302208:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828671.711452:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828671.711465:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828671/real 1787828671] req@ffff976889eff480 x1874670564302208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828671.711475:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564302208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828671.711482:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828671.711484:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828671.711487:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564302208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828671.711492:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564302336:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828671.711507:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828671.711513:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828671/real 1787828671] req@ffff976889efe580 x1874670564302336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828671.711521:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564302336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828671.711532:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828671.711536:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828671.711539:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564302336:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828671.711546:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564302464:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828671.711569:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828671.711598:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828671/real 1787828671] req@ffff976889efe1c0 x1874670564302464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828671.711607:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564302464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828671.711613:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828671.711616:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828671.711618:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564302464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828671.711623:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564302592:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828671.711639:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828671.711646:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828671/real 1787828671] req@ffff976889efcb40 x1874670564302592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828671.711654:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564302592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828671.711660:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828671.711662:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828671.711665:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564302592:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828671.966929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828671.966937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828671.966996:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828672.734941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828672.990959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828672.991020:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828672.991032:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828673.759054:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828674.014912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828674.014929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828674.015001:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828674.782912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828675.038933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828675.039006:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828675.039027:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828675.806900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828676.062922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828676.062922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828676.063030:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828676.830961:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828676.830981:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828676.830983:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828676.830989:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828676.831003:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3616 00000100:00080000:2.0:1787828676.831015:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828676.831022:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828676.831065:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564302720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828676.831109:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828676.831114:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828676.831116:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828676.831122:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3616 00000100:00080000:2.0:1787828676.831126:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828676.831131:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828676.831145:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828676.831162:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564302848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828676.831176:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564302720:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828676.831178:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828676.831193:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828676.831199:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828676.831218:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3616 00000100:00080000:2.0:1787828676.831222:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828676.831225:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828676.831230:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828676.831241:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564302976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828676.831249:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828676.831253:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828676.831256:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828676.831262:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3616 00000100:00080000:2.0:1787828676.831267:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828676.831272:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828676.831290:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1787828676.831302:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828676.831308:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564303104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828676.831320:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828676.831323:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828676.831325:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828676.831329:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3616 00000100:00080000:2.0:1787828676.831333:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828676.831337:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828676.831343:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828676.831346:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828676/real 1787828676] req@ffff976889cfa940 x1874670564302720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828676.831375:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564303232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828676.831389:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564302720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828676.831416:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828676.831439:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828676.831447:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564302720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828676.831490:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564302848:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828676.831530:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828676.831575:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828676/real 1787828676] req@ffff976889cfad00 x1874670564302848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828676.831605:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564302848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828676.831611:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828676.831614:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828676.831616:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564302848:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828676.831621:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564302976:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828676.831635:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828676.831642:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828676/real 1787828676] req@ffff976889cf92c0 x1874670564302976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828676.831666:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564302976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828676.831680:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828676.831686:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828676.831691:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564302976:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828676.831702:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564303104:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828676.831762:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828676.831778:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828676/real 1787828676] req@ffff976889cf9e00 x1874670564303104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828676.831804:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564303104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828676.831810:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828676.831812:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828676.831814:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564303104:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828676.831818:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564303232:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828676.831832:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828676.831877:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828676/real 1787828676] req@ffff976889cfb480 x1874670564303232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828676.831889:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564303232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828676.831896:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828676.831899:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828676.831902:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564303232:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828677.086921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828677.086921:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828677.087006:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828677.854962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828678.110912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828678.110952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828678.111005:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828678.878917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828679.134908:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828679.134908:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828679.135003:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828679.902938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828680.158894:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828680.158960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828680.158969:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828680.926951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828681.182921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828681.182921:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828681.183012:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828681.950934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828681.950935:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828681.950954:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828681.950958:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828681.950967:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3621 00000100:00080000:2.0:1787828681.950976:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828681.950982:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828681.951026:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564303360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828681.951044:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828681.951047:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828681.951048:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828681.951051:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3621 00000100:00080000:2.0:1787828681.951053:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828681.951056:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828681.951066:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828681.951076:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564303360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828681.951079:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564303488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828681.951089:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828681.951092:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828681.951093:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828681.951095:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3621 00000100:00080000:2.0:1787828681.951097:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828681.951099:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828681.951102:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828681.951111:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564303616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828681.951125:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828681.951127:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828681.951128:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828681.951131:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3621 00000100:00080000:2.0:1787828681.951133:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828681.951135:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828681.951137:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828681.951143:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564303744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828681.951149:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828681.951151:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828681.951154:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828681.951155:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828681.951159:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3621 00000100:00080000:2.0:1787828681.951161:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828681.951164:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828681.951168:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828681.951177:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564303872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828681.951195:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828681/real 1787828681] req@ffff976889cfa1c0 x1874670564303360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828681.951208:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564303360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828681.951221:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828681.951224:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828681.951227:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564303360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828681.951245:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564303488:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828681.951261:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828681.951269:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828681/real 1787828681] req@ffff976889cfb480 x1874670564303488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828681.951277:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564303488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828681.951283:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828681.951285:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828681.951287:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564303488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828681.951291:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564303616:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828681.951302:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828681.951307:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828681/real 1787828681] req@ffff976889cf9e00 x1874670564303616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828681.951315:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564303616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828681.951320:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828681.951321:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828681.951323:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564303616:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828681.951326:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564303744:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828681.951336:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828681.951341:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828681/real 1787828681] req@ffff976889cf92c0 x1874670564303744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828681.951348:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564303744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828681.951364:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828681.951366:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828681.951367:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564303744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828681.951371:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564303872:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828681.951380:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828681.951385:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828681/real 1787828681] req@ffff976889cfad00 x1874670564303872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828681.951392:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564303872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828681.951397:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828681.951399:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828681.951401:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564303872:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828682.206926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828682.206926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828682.207011:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828682.974897:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828683.230916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828683.230929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828683.230978:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828683.998927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828684.254947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828684.254947:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828684.255020:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828685.022994:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828685.278902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828685.278921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828685.278972:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828686.046949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828686.302929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828686.302929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828686.302995:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828687.070950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828687.070972:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828687.070980:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828687.070986:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828687.070997:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3626 00000100:00080000:2.0:1787828687.071026:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828687.071032:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828687.071083:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564304000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828687.071114:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828687.071120:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828687.071122:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828687.071128:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3626 00000100:00080000:2.0:1787828687.071133:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828687.071148:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828687.071161:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828687.071167:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564304000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828687.071184:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564304128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828687.071213:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828687.071216:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828687.071218:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828687.071222:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3626 00000100:00080000:2.0:1787828687.071227:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828687.071230:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828687.071236:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828687.071238:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828687.071252:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564304256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828687.071262:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828687.071265:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828687.071266:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828687.071270:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3626 00000100:00080000:2.0:1787828687.071274:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828687.071278:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828687.071283:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828687.071285:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828687/real 1787828687] req@ffff976889cfa940 x1874670564304000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828687.071315:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564304384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828687.071330:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828687.071334:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828687.071336:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828687.071342:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3626 00000100:00080000:2.0:1787828687.071347:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828687.071356:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828687.071358:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564304000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828687.071362:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828687.071383:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564304512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828687.071384:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828687.071401:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828687.071412:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564304000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828687.071445:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564304128:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828687.071503:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828687.071541:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828687/real 1787828687] req@ffff976889cf8f00 x1874670564304128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828687.071562:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564304128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828687.071577:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828687.071583:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828687.071589:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564304128:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828687.071599:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564304256:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828687.071644:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828687.071656:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828687/real 1787828687] req@ffff976889cf8000 x1874670564304256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828687.071667:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564304256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828687.071674:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828687.071677:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828687.071679:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564304256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828687.071712:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564304384:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828687.071735:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828687.071746:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828687/real 1787828687] req@ffff976889cfb0c0 x1874670564304384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828687.071759:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564304384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828687.071767:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828687.071770:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828687.071773:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564304384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828687.071779:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564304512:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828687.071793:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828687.071802:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828687/real 1787828687] req@ffff976889cfa580 x1874670564304512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828687.071813:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564304512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828687.071821:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828687.071824:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828687.071827:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564304512:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828687.326957:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828687.326960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828687.327099:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828688.094908:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828688.350961:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828688.350993:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828688.351190:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828689.118926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828689.374909:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828689.374909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828689.374978:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828690.142930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828690.398891:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828690.398900:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828690.398925:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828691.166918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828691.422886:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828691.422892:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828691.422934:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828692.190907:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828692.190907:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828692.190915:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828692.190920:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828692.190929:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3632 00000100:00080000:2.0:1787828692.190939:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828692.190946:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828692.190988:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564304640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828692.191010:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828692.191014:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828692.191016:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828692.191020:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3632 00000100:00080000:2.0:1787828692.191024:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828692.191028:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828692.191038:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828692.191048:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564304640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828692.191052:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564304768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828692.191062:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828692.191065:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828692.191066:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828692.191069:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3632 00000100:00080000:2.0:1787828692.191072:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828692.191076:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828692.191080:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828692.191089:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564304896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828692.191097:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828692.191100:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787828692.191102:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828692.191102:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828692.191106:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3632 00000100:00080000:2.0:1787828692.191108:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828692.191112:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828692.191116:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828692.191134:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564305024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828692.191138:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828692/real 1787828692] req@ffff976889cfbc00 x1874670564304640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828692.191145:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828692.191148:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828692.191150:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828692.191153:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3632 00000100:00100000:1.0:1787828692.191154:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564304640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828692.191157:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828692.191160:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:1.0:1787828692.191165:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787828692.191165:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828692.191169:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828692.191173:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564304640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828692.191174:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564305152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828692.191195:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564304768:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828692.191219:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828692.191233:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828692/real 1787828692] req@ffff976889cfa580 x1874670564304768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828692.191246:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564304768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828692.191255:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828692.191259:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828692.191262:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564304768:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828692.191269:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564304896:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828692.191286:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828692.191294:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828692/real 1787828692] req@ffff976889cfb0c0 x1874670564304896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828692.191303:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564304896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828692.191308:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828692.191310:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828692.191312:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564304896:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828692.191315:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564305024:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828692.191326:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828692.191342:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828692/real 1787828692] req@ffff976889cf8000 x1874670564305024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828692.191350:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564305024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828692.191355:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828692.191356:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828692.191358:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564305024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828692.191361:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564305152:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828692.191371:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828692.191376:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828692/real 1787828692] req@ffff976889cf8f00 x1874670564305152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828692.191383:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564305152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828692.191393:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828692.191395:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828692.191397:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564305152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828692.446875:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828692.446898:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828692.446964:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828693.214904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828693.470900:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828693.470905:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828693.470953:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828694.238923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828694.494903:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828694.494914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828694.495009:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828695.262963:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828695.518915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828695.518940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828695.519051:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828696.286921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828696.543011:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828696.543011:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828696.543045:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828697.310991:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828697.310997:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828697.311003:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828697.311009:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828697.311019:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3637 00000100:00080000:3.0:1787828697.311030:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828697.311038:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828697.311076:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564305280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828697.311114:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828697.311118:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828697.311119:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828697.311123:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3637 00000100:00080000:3.0:1787828697.311126:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828697.311129:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828697.311138:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828697.311148:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564305408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828697.311158:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828697.311161:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828697.311163:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828697.311166:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3637 00000100:00080000:3.0:1787828697.311168:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828697.311171:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828697.311176:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828697.311182:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564305280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828697.311216:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564305536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828697.311232:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828697.311236:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828697.311238:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828697.311243:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3637 00000100:00080000:3.0:1787828697.311266:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828697.311271:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828697.311277:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1787828697.311295:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828697.311295:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564305664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828697.311310:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828697.311315:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828697.311317:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828697.311322:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3637 00000100:00080000:3.0:1787828697.311326:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828697.311329:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828697.311334:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828697.311358:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564305792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828697.311359:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828697/real 1787828697] req@ffff97688cb5f0c0 x1874670564305280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828697.311423:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564305280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828697.311441:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828697.311449:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828697.311455:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564305280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828697.311468:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564305408:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828697.311500:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828697.311522:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828697/real 1787828697] req@ffff97688cb5d2c0 x1874670564305408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828697.311557:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564305408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828697.311572:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828697.311578:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828697.311583:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564305408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828697.311643:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564305536:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828697.311666:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828697.311695:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828697/real 1787828697] req@ffff97688cb5e1c0 x1874670564305536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828697.311706:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564305536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828697.311713:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828697.311716:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828697.311719:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564305536:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828697.311724:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564305664:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828697.311738:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828697.311746:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828697/real 1787828697] req@ffff97688cb5c780 x1874670564305664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828697.311755:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564305664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828697.311768:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828697.311770:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828697.311773:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564305664:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828697.311777:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564305792:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828697.311789:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828697.311796:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828697/real 1787828697] req@ffff97688cb5c3c0 x1874670564305792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828697.311805:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564305792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828697.311811:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828697.311813:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828697.311816:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564305792:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828697.566905:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828697.566906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828697.566967:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828698.334932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828698.590964:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828698.591058:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828698.591164:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828699.358977:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828699.614992:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828699.614998:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828699.615094:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828700.382891:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828700.638925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828700.638925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828700.639076:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828701.406991:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828701.662974:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828701.662993:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828701.662994:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828702.430935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828702.430980:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828702.430986:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828702.430991:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828702.431000:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3642 00000100:00080000:2.0:1787828702.431010:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828702.431017:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828702.431049:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564305920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828702.431075:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828702.431078:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828702.431079:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828702.431082:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3642 00000100:00080000:2.0:1787828702.431086:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828702.431089:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828702.431097:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828702.431116:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564305920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828702.431124:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564306048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828702.431136:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828702.431139:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828702.431140:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828702.431143:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3642 00000100:00080000:2.0:1787828702.431146:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828702.431151:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828702.431155:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828702.431165:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564306176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828702.431172:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828702.431176:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828702.431177:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828702.431182:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828702.431189:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3642 00000100:00080000:2.0:1787828702.431197:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828702.431203:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828702.431208:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828702.431224:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564306304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828702.431232:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828702/real 1787828702] req@ffff976889cfa940 x1874670564305920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828702.431233:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828702.431241:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828702.431245:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828702.431254:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3642 00000100:00080000:2.0:1787828702.431258:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828702.431260:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828702.431264:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564305920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828702.431265:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787828702.431278:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828702.431279:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564306432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828702.431287:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828702.431388:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564305920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828702.431425:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564306048:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828702.431581:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828702.431642:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828702/real 1787828702] req@ffff976889cfad00 x1874670564306048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828702.431662:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564306048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828702.431671:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828702.431676:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828702.431686:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564306048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828702.431692:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564306176:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828702.431710:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828702.431718:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828702/real 1787828702] req@ffff976889cf92c0 x1874670564306176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828702.431725:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564306176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828702.431731:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828702.431733:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828702.431735:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564306176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828702.431739:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564306304:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828702.431749:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828702.431756:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828702/real 1787828702] req@ffff976889cf9e00 x1874670564306304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828702.431768:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564306304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828702.431777:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828702.431780:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828702.431784:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564306304:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828702.431790:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564306432:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828702.431811:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828702.431819:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828702/real 1787828702] req@ffff976889cfb480 x1874670564306432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828702.431853:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564306432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828702.431859:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828702.431861:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828702.431863:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564306432:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828702.686949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828702.687032:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828702.687131:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828703.454956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828703.710965:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828703.711001:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828703.711009:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828704.478948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828704.734961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828704.734972:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828704.735118:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828705.502931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828705.758960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828705.758960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828705.758998:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828706.526964:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828706.782956:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828706.782962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828706.783031:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828707.551020:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828707.551024:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828707.551032:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828707.551037:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828707.551046:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3647 00000100:00080000:3.0:1787828707.551056:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828707.551067:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828707.551124:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564306560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828707.551156:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828707.551159:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828707.551161:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828707.551166:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3647 00000100:00080000:3.0:1787828707.551170:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828707.551173:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828707.551183:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828707.551207:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564306688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828707.551217:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828707.551221:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828707.551223:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828707.551226:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3647 00000100:00080000:3.0:1787828707.551228:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828707.551232:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828707.551236:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828707.551246:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564306816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828707.551256:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828707.551259:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828707.551261:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828707.551265:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3647 00000100:00080000:3.0:1787828707.551268:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828707.551272:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828707.551276:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828707.551285:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564306560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828707.551287:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564306944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828707.551305:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828707.551308:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828707.551310:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828707.551315:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3647 00000100:00080000:3.0:1787828707.551318:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828707.551339:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828707.551344:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828707.551377:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564307072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828707.551445:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564306688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828707.551475:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564306816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828707.551504:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564306944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828707.551524:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564307072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828707.806947:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828707.806969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828707.806993:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828708.575012:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828708.830951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828708.830962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828708.830983:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828709.599044:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828709.854949:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828709.854981:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828709.854982:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828710.622909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:1.0:1787828710.623122:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:1.0:1787828710.623160:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787828710.623173:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787828710.623205:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787828710.623211:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787828710.623216:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787828710.623223:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:3.0:1787828710.623251:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828707/real 1787828710] req@ffff97688cb5d2c0 x1874670564306944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828710.623276:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564306944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828710.623294:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828710.623316:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828710.623321:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564306944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828710.623329:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828707/real 1787828710] req@ffff97688cb5f0c0 x1874670564307072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828710.623342:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564307072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828710.623350:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828710.623353:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828710.623357:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564307072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828710.623364:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828707/real 1787828710] req@ffff97688cb5c3c0 x1874670564306560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828710.623376:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564306560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828710.623383:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828710.623386:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828710.623389:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564306560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828710.623396:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828707/real 1787828710] req@ffff97688cb5c780 x1874670564306688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828710.623407:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564306688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828710.623415:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828710.623419:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828710.623422:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564306688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828710.623428:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828707/real 1787828710] req@ffff97688cb5e1c0 x1874670564306816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828710.623441:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564306816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828710.623448:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828710.623451:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828710.623455:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564306816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828710.878908:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828710.878915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828710.878990:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828711.646926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828711.902905:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828711.902927:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828711.903099:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828712.670932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828712.671093:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828712.671104:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828712.671110:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828712.671125:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3652 00000100:00080000:2.0:1787828712.671143:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828712.671155:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828712.671210:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564307200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828712.671260:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828712.671265:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828712.671268:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828712.671275:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3652 00000100:00080000:2.0:1787828712.671282:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828712.671309:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828712.671326:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828712.671337:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564307200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828712.671356:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564307328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828712.671376:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828712.671382:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828712.671385:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828712.671393:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3652 00000100:00080000:2.0:1787828712.671400:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828712.671407:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828712.671416:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828712.671437:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564307456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828712.671453:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828712.671457:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828712.671461:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828712.671466:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828712.671475:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3652 00000100:00080000:2.0:1787828712.671481:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828712.671487:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828712.671495:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828712.671510:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564307584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828712.671525:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828712.671530:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828712.671533:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828712.671539:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3652 00000100:00080000:2.0:1787828712.671546:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828712.671554:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828712/real 1787828712] req@ffff976889cfa1c0 x1874670564307200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828712.671557:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828712.671570:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828712.671595:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564307712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828712.671598:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564307200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828712.671651:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828712.671658:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828712.671666:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564307200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828712.671709:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564307328:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828712.671763:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828712.671784:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828712/real 1787828712] req@ffff976889cfb480 x1874670564307328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828712.672245:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564307328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828712.672257:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828712.672261:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828712.672265:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564307328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828712.672273:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564307456:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828712.672317:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828712.672331:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828712/real 1787828712] req@ffff976889cf9e00 x1874670564307456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828712.672343:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564307456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828712.672353:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828712.672356:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828712.672360:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564307456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828712.672367:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564307584:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828712.672390:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828712.672402:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828712/real 1787828712] req@ffff976889cf92c0 x1874670564307584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828712.672423:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564307584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828712.672432:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828712.672435:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828712.672439:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564307584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828712.672446:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564307712:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828712.672469:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828712.672481:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828712/real 1787828712] req@ffff976889cfad00 x1874670564307712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828712.672493:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564307712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828712.672502:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828712.672506:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828712.672510:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564307712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828712.926992:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828712.926992:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828712.927062:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828713.694943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828713.950994:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828713.950994:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828713.951013:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828714.718951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828714.974925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828714.974927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828714.975041:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828715.742927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828715.998923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828715.998923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828715.998969:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828716.766955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828717.022920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828717.022921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828717.023016:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828717.790939:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828717.790951:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828717.790957:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828717.790969:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3657 00000100:00100000:1.0:1787828717.790971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828717.790985:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828717.790996:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828717.791046:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564307840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828717.791082:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828717.791086:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828717.791088:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828717.791093:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3657 00000100:00080000:3.0:1787828717.791097:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828717.791104:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828717.791116:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828717.791121:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564307840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828717.791137:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564307968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828717.791152:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828717.791157:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828717.791159:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828717.791164:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3657 00000100:00080000:3.0:1787828717.791169:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828717.791174:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828717.791181:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828717.791187:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828717.791198:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564308096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828717.791212:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828717.791216:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828717.791219:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828717.791228:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3657 00000100:00080000:3.0:1787828717.791233:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828717.791240:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828717/real 1787828717] req@ffff97688cb5e1c0 x1874670564307840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828717.791240:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828717.791264:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828717.791283:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564307840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828717.791285:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564308224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828717.791320:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828717.791324:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828717.791327:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:3.0:1787828717.791334:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787828717.791338:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564307840:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787828717.791340:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828717.791348:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3657 00000100:00080000:3.0:1787828717.791353:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828717.791358:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828717.791364:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828717.791368:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564307968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828717.791389:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564308352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828717.791402:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828717.791423:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828717/real 1787828717] req@ffff97688cb5c780 x1874670564307968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828717.791433:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564307968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828717.791439:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828717.791442:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828717.791445:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564307968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828717.791450:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564308096:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828717.791462:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828717.791468:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828717/real 1787828717] req@ffff97688cb5c3c0 x1874670564308096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828717.791475:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564308096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828717.791480:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828717.791482:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828717.791483:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564308096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828717.791487:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564308224:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828717.791498:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828717.791504:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828717/real 1787828717] req@ffff97688cb5f0c0 x1874670564308224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828717.791511:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564308224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828717.791516:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828717.791518:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828717.791519:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564308224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828717.791533:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564308352:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828717.791544:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828717.791549:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828717/real 1787828717] req@ffff97688cb5d2c0 x1874670564308352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828717.791561:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564308352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828717.791567:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828717.791568:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828717.791570:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564308352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828718.046942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828718.046948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828718.046981:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828718.814952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828719.070898:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828719.070940:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828719.070951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828719.838928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828720.094953:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828720.094957:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828720.095030:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828720.862967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828721.118946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828721.118979:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828721.118979:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828721.886922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828722.142922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828722.142932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828722.142960:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828722.910895:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828722.910911:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1787828722.910914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828722.910915:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828722.910927:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3662 00000100:00080000:3.0:1787828722.910938:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828722.910946:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828722.910983:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564308480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828722.911009:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828722.911012:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828722.911013:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828722.911028:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3662 00000100:00080000:3.0:1787828722.911032:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828722.911036:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828722.911050:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828722.911051:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564308480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828722.911064:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564308608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828722.911073:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828722.911075:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828722.911077:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828722.911080:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3662 00000100:00080000:3.0:1787828722.911092:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828722.911094:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828722.911097:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828722.911105:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564308736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828722.911112:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828722.911115:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828722.911116:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828722.911120:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3662 00000800:00000100:1.0:1787828722.911125:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787828722.911126:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828722.911132:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828722.911135:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828722.911144:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564308864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828722.911150:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828722.911152:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828722.911153:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828722.911156:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3662 00000100:00080000:3.0:1787828722.911158:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828722.911161:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828722.911164:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828722.911167:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828722/real 1787828722] req@ffff97688cb5d680 x1874670564308480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828722.911180:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564308992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828722.911185:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564308480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828722.911198:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828722.911201:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828722.911205:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564308480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828722.911236:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564308608:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828722.911252:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828722.911272:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828722/real 1787828722] req@ffff97688cb5c000 x1874670564308608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828722.911280:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564308608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828722.911286:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828722.911288:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828722.911291:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564308608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828722.911294:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564308736:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828722.911304:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828722.911310:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828722/real 1787828722] req@ffff97688cb5cb40 x1874670564308736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828722.911318:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564308736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828722.911333:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828722.911334:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828722.911336:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564308736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828722.911339:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564308864:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828722.911349:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828722.911355:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828722/real 1787828722] req@ffff97688cb5de00 x1874670564308864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828722.911361:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564308864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828722.911367:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828722.911369:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828722.911371:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564308864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828722.911374:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564308992:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828722.911383:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828722.911388:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828722/real 1787828722] req@ffff97688cb5f840 x1874670564308992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828722.911395:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564308992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828722.911399:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828722.911401:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828722.911403:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564308992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828723.166946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828723.166946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828723.166998:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828723.934934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828724.190903:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828724.190910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828724.190947:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828724.958987:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828725.214925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828725.214925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828725.214948:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828725.982949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828726.238912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828726.238913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828726.239015:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828727.006996:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828727.262940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828727.262940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828727.262976:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828728.030919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828728.030937:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828728.030945:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828728.030949:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828728.030959:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3667 00000100:00080000:1.0:1787828728.030981:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828728.030988:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828728.031026:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564309120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828728.031065:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828728.031070:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828728.031072:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828728.031078:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3667 00000100:00080000:1.0:1787828728.031082:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828728.031088:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828728.031098:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564309120:192.168.203.139@tcp:250:kworker.0 00000008:00100000:1.0:1787828728.031099:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828728.031119:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564309248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828728.031131:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828728.031135:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828728.031137:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828728.031141:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3667 00000100:00080000:1.0:1787828728.031146:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828728.031150:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:3.0:1787828728.031156:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787828728.031159:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828728.031176:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564309376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828728.031205:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828728/real 1787828728] req@ffff976889efe940 x1874670564309120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828728.031219:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828728.031225:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828728.031227:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828728.031238:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3667 00000100:00080000:1.0:1787828728.031247:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828728.031253:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828728.031256:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564309120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828728.031261:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1787828728.031269:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828728.031287:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828728.031291:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564309504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828728.031296:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564309120:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787828728.031313:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828728.031317:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828728.031319:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828728.031324:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3667 00000100:00100000:3.0:1787828728.031328:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564309248:192.168.203.139@tcp:38:kworker.0 00000100:00080000:1.0:1787828728.031329:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828728.031338:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828728.031344:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787828728.031357:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828728.031363:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564309632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828728.031370:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828728/real 1787828728] req@ffff976889eff0c0 x1874670564309248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828728.031389:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564309248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828728.031397:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828728.031399:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828728.031402:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564309248:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828728.031408:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564309376:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828728.031420:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828728.031426:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828728/real 1787828728] req@ffff976889effc00 x1874670564309376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828728.031442:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564309376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828728.031452:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828728.031454:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828728.031457:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564309376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828728.031460:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564309504:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828728.031472:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828728.031478:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828728/real 1787828728] req@ffff976889efed00 x1874670564309504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828728.031485:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564309504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828728.031490:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828728.031492:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828728.031494:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564309504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828728.031508:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564309632:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828728.031524:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828728.031531:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828728/real 1787828728] req@ffff976889efc3c0 x1874670564309632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828728.031539:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564309632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828728.031544:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828728.031546:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828728.031548:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564309632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828728.286909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828728.286910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828728.286987:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828729.054944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828729.310932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828729.310954:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828729.311000:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828730.078901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828730.334913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828730.334913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828730.334969:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828731.102946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828731.358935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828731.358936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828731.358960:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828732.126893:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828732.382925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828732.382976:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828732.382985:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828733.150928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828733.150935:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828733.150944:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828733.150948:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828733.150959:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3673 00000100:00080000:3.0:1787828733.150970:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828733.150978:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828733.151017:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564309760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828733.151049:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828733.151052:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828733.151054:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828733.151057:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3673 00000100:00080000:3.0:1787828733.151061:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828733.151065:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828733.151075:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828733.151082:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564309760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828733.151088:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564309888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828733.151097:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828733.151104:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828733.151106:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828733.151108:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3673 00000100:00080000:3.0:1787828733.151112:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828733.151115:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828733.151118:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828733.151126:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564310016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828733.151133:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828733.151135:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828733.151138:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828733.151139:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787828733.151155:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3673 00000100:00080000:3.0:1787828733.151159:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828733.151162:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828733.151167:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828733.151177:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564310144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828733.151184:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828733.151187:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828733.151189:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828733.151193:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828733/real 1787828733] req@ffff97688cb5f840 x1874670564309760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828733.151194:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3673 00000100:00080000:3.0:1787828733.151198:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828733.151203:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828733.151210:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828733.151219:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564309760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828733.151221:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564310272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828733.151265:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828733.151270:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828733.151275:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564309760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828733.151297:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564309888:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828733.151313:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828733.151322:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828733/real 1787828733] req@ffff97688cb5de00 x1874670564309888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828733.151331:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564309888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828733.151336:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828733.151339:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828733.151341:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564309888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828733.151345:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564310016:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828733.151355:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828733.151361:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828733/real 1787828733] req@ffff97688cb5cb40 x1874670564310016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828733.151369:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564310016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828733.151374:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828733.151376:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828733.151377:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564310016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828733.151381:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564310144:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828733.151391:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828733.151397:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828733/real 1787828733] req@ffff97688cb5c000 x1874670564310144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828733.151403:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564310144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828733.151415:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828733.151417:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828733.151419:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564310144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828733.151422:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564310272:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828733.151433:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828733.151439:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828733/real 1787828733] req@ffff97688cb5d680 x1874670564310272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828733.151446:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564310272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828733.151451:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828733.151453:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828733.151455:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564310272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828733.406945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828733.406975:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828733.407007:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828734.174901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828734.430913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828734.430950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828734.431041:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828735.198913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828735.454980:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828735.455002:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828735.455050:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828736.222978:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828736.478934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828736.478941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828736.478971:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828737.246908:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828737.502902:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828737.502902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828737.503002:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828738.270970:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828738.270982:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828738.270988:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828738.270991:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828738.271000:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3678 00000100:00080000:0.0:1787828738.271012:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828738.271020:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828738.271057:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564310400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828738.271084:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828738.271095:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828738.271098:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828738.271107:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3678 00000100:00100000:1.0:1787828738.271122:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564310400:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828738.271125:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828738.271133:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828738.271144:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828738.271160:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564310528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828738.271170:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828738.271174:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828738.271176:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828738.271183:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3678 00000800:00000100:1.0:1787828738.271183:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828738.271191:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828738.271211:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828738.271216:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828738.271228:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564310656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828738.271244:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828738.271248:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828738.271251:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828738.271253:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828738/real 1787828738] req@ffff976882aed680 x1874670564310400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828738.271276:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3678 00000100:00080000:0.0:1787828738.271284:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828738.271291:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828738.271292:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564310400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828738.271296:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1787828738.271307:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828738.271310:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828738.271314:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564310784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828738.271317:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564310400:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828738.271332:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828738.271335:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828738.271336:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828738.271340:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3678 00000100:00080000:0.0:1787828738.271342:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828738.271345:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828738.271348:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828738.271352:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564310528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828738.271358:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564310912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828738.271386:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828738.271397:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828738/real 1787828738] req@ffff976882aee940 x1874670564310528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828738.271415:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564310528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828738.271421:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828738.271424:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828738.271426:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564310528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828738.271430:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564310656:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828738.271442:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828738.271449:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828738/real 1787828738] req@ffff976882aef0c0 x1874670564310656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828738.271456:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564310656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828738.271462:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828738.271474:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828738.271477:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564310656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828738.271481:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564310784:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828738.271492:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828738.271497:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828738/real 1787828738] req@ffff976882aeed00 x1874670564310784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828738.271505:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564310784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828738.271510:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828738.271512:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828738.271513:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564310784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828738.271526:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564310912:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828738.271539:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828738.271547:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828738/real 1787828738] req@ffff976882aee580 x1874670564310912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828738.271562:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564310912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828738.271569:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828738.271572:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828738.271574:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564310912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828738.526882:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828738.526897:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828738.526932:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828739.294947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828739.550950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828739.550950:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828739.550991:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828740.318953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828740.574903:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828740.574924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828740.575014:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828741.342901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828741.598919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828741.598919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828741.599045:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828742.366942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828742.622950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828742.622950:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828742.623011:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828743.390921:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828743.390922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828743.390934:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828743.390940:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828743.390954:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3683 00000100:00080000:0.0:1787828743.390965:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828743.390972:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828743.391029:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564311040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828743.391068:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828743.391118:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828743.391120:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828743.391125:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3683 00000100:00080000:0.0:1787828743.391128:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828743.391133:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828743.391143:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828743.391155:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564311040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828743.391161:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564311168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828743.391172:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828743.391175:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828743.391177:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828743.391180:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3683 00000100:00080000:0.0:1787828743.391183:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828743.391185:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828743.391189:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828743.391198:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564311296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828743.391220:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828743.391227:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828743.391231:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828743.391233:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828743.391237:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3683 00000100:00080000:0.0:1787828743.391243:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828743.391246:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828743.391251:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828743.391265:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828743/real 1787828743] req@ffff976882aec3c0 x1874670564311040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828743.391269:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564311424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828743.391290:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564311040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828743.391291:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828743.391304:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828743.391309:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828743.391309:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828743.391321:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:0.0:1787828743.391322:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3683 00000100:00100000:1.0:1787828743.391328:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564311040:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828743.391331:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828743.391339:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828743.391345:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828743.391352:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564311168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828743.391363:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564311552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828743.391375:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828743.391389:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828743/real 1787828743] req@ffff976882aee580 x1874670564311168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828743.391401:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564311168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828743.391408:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828743.391410:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828743.391413:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564311168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828743.391417:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564311296:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828743.391429:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828743.391436:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828743/real 1787828743] req@ffff976882aeed00 x1874670564311296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828743.391445:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564311296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828743.391451:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828743.391453:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828743.391455:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564311296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828743.391459:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564311424:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828743.391470:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828743.391480:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828743/real 1787828743] req@ffff976882aef0c0 x1874670564311424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828743.391494:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564311424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828743.391505:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828743.391509:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828743.391514:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564311424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828743.391534:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564311552:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828743.391556:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828743.391565:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828743/real 1787828743] req@ffff976882aee940 x1874670564311552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828743.391574:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564311552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828743.391581:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828743.391583:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828743.391586:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564311552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828743.646946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828743.646948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828743.647063:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828744.415000:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828744.670958:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828744.670989:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828744.671063:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828745.438965:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828745.694933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828745.694934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828745.695075:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828746.462946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828746.719024:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828746.719024:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828746.719156:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828747.486996:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828747.742929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828747.742929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828747.743016:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828748.510984:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828748.510987:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828748.511001:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828748.511006:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828748.511018:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3688 00000100:00080000:0.0:1787828748.511031:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828748.511070:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828748.511130:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564311680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828748.511155:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828748.511157:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828748.511159:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828748.511162:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3688 00000100:00080000:0.0:1787828748.511165:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828748.511168:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828748.511177:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828748.511184:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564311680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828748.511190:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564311808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828748.511199:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828748.511202:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828748.511203:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828748.511206:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3688 00000100:00080000:0.0:1787828748.511208:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828748.511210:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828748.511213:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828748.511223:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564311936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828748.511253:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828748.511258:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828748.511260:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828748.511267:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828748.511272:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3688 00000100:00080000:0.0:1787828748.511282:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828748.511290:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828748.511314:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828748.511329:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828748/real 1787828748] req@ffff976882aed680 x1874670564311680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828748.511332:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564312064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828748.511354:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828748.511358:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564311680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828748.511359:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828748.511363:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828748.511371:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3688 00000100:00080000:0.0:1787828748.511377:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828748.511377:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828748.511384:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:1.0:1787828748.511385:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828748.511393:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564311680:192.168.203.139@tcp:250:kworker.0 00000008:00100000:0.0:1787828748.511394:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828748.511407:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564312192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828748.511418:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564311808:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828748.511436:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828748.511466:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828748/real 1787828748] req@ffff976882aee940 x1874670564311808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828748.511478:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564311808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828748.511488:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828748.511493:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828748.511498:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564311808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828748.511522:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564311936:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828748.511541:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828748.511553:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828748/real 1787828748] req@ffff976882aef0c0 x1874670564311936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828748.511572:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564311936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828748.511577:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828748.511579:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828748.511581:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564311936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828748.511586:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564312064:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828748.511619:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828748.511648:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828748/real 1787828748] req@ffff976882aeed00 x1874670564312064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828748.511674:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564312064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828748.511703:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828748.511707:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828748.511711:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564312064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828748.511725:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564312192:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828748.511741:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828748.511748:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828748/real 1787828748] req@ffff976882aee580 x1874670564312192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828748.511756:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564312192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828748.511765:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828748.511767:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828748.511769:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564312192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828748.766920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828748.766920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828748.766976:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828749.534928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828749.790951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828749.790977:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828749.790984:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828750.558949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828750.814957:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828750.814958:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828750.815063:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828751.582894:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828751.838942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828751.838942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828751.838998:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828752.606960:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828752.862985:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828752.862985:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828752.863094:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828753.630956:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828753.630956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828753.630970:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828753.630975:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828753.630986:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3693 00000100:00080000:0.0:1787828753.630997:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828753.631022:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828753.631101:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564312320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828753.631149:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828753.631153:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828753.631156:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828753.631179:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3693 00000100:00080000:0.0:1787828753.631184:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828753.631189:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828753.631201:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828753.631208:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564312320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828753.631223:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564312448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828753.631236:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828753.631240:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828753.631243:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828753.631246:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3693 00000100:00080000:0.0:1787828753.631278:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828753.631282:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828753.631287:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828753.631290:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828753.631306:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564312576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828753.631318:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828753.631321:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828753.631322:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828753.631326:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3693 00000100:00080000:0.0:1787828753.631331:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828753.631346:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828753.631353:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828753.631354:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828753/real 1787828753] req@ffff976882aec3c0 x1874670564312320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828753.631379:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564312704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828753.631382:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564312320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828753.631426:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828753.631430:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828753.631433:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:0.0:1787828753.631438:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828753.631444:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828753.631445:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564312320:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828753.631454:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3693 00000100:00080000:0.0:1787828753.631461:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828753.631465:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828753.631481:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828753.631491:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564312448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828753.631499:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564312832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828753.631525:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828753.631544:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828753/real 1787828753] req@ffff976882aee580 x1874670564312448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828753.631557:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564312448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828753.631574:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828753.631577:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828753.631581:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564312448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828753.631586:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564312576:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828753.631601:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828753.631611:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828753/real 1787828753] req@ffff976882aeed00 x1874670564312576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828753.631622:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564312576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828753.631629:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828753.631631:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828753.631634:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564312576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828753.631639:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564312704:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828753.631654:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828753.631669:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828753/real 1787828753] req@ffff976882aef0c0 x1874670564312704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828753.631707:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564312704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828753.631733:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828753.631738:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828753.631743:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564312704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828753.631767:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564312832:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828753.631790:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828753.631801:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828753/real 1787828753] req@ffff976882aee940 x1874670564312832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828753.631819:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564312832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828753.631873:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828753.631879:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828753.631884:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564312832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828753.886975:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828753.886975:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828753.887082:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828754.654922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828754.910971:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828754.910987:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828754.911030:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828755.678968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828755.935085:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828755.935094:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828755.935138:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828756.702902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828756.958876:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828756.958892:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828756.958927:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828757.726920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828757.982873:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828757.982895:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828757.982933:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828758.750914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828758.751010:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828758.751015:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828758.751018:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828758.751026:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3698 00000100:00080000:2.0:1787828758.751034:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828758.751040:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828758.751074:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564312960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828758.751101:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828758.751105:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828758.751107:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828758.751112:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3698 00000100:00080000:2.0:1787828758.751117:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828758.751123:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828758.751133:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564312960:192.168.203.139@tcp:250:kworker.0 00000008:00100000:2.0:1787828758.751134:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828758.751152:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564313088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828758.751160:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828758.751163:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828758.751165:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828758.751167:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3698 00000100:00080000:2.0:1787828758.751170:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828758.751172:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828758.751176:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828758.751184:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564313216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828758.751191:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828758.751194:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828758.751195:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828758.751197:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3698 00000100:00080000:2.0:1787828758.751200:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828758.751202:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828758.751205:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1787828758.751210:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828758.751214:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564313344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828758.751222:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828758.751224:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828758.751226:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828758.751229:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3698 00000100:00080000:2.0:1787828758.751232:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828758.751234:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828758.751237:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828758.751245:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564313472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828758.751252:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828758/real 1787828758] req@ffff976889cfa940 x1874670564312960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828758.751269:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564312960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828758.751277:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828758.751280:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828758.751283:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564312960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828758.751301:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564313088:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828758.751317:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828758.751326:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828758/real 1787828758] req@ffff976889cf8f00 x1874670564313088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828758.751334:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564313088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828758.751339:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828758.751342:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828758.751344:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564313088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828758.751347:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564313216:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828758.751357:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828758.751363:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828758/real 1787828758] req@ffff976889cf8000 x1874670564313216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828758.751375:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564313216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828758.751380:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828758.751381:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828758.751383:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564313216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828758.751386:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564313344:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828758.751396:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828758.751413:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828758/real 1787828758] req@ffff976889cfb0c0 x1874670564313344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828758.751421:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564313344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828758.751427:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828758.751429:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828758.751431:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564313344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828758.751434:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564313472:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828758.751445:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828758.751451:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828758/real 1787828758] req@ffff976889cfa580 x1874670564313472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828758.751458:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564313472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828758.751467:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828758.751469:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828758.751470:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564313472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828759.006870:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828759.006914:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828759.006929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828759.774940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787828760.030923:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828760.030954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828760.030955:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828760.798905:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828761.054971:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828761.054990:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828761.055037:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828761.822937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828762.078931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828762.078980:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828762.079004:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828762.847072:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828763.102932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787828763.102932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828763.103200:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828763.871026:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828763.871043:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828763.871051:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828763.871063:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828763.871069:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3703 00000100:00080000:2.0:1787828763.871090:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828763.871104:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828763.871157:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564313600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828763.871199:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828763.871206:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828763.871209:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828763.871216:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3703 00000100:00080000:2.0:1787828763.871223:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828763.871231:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828763.871247:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828763.871282:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564313600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828763.871283:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564313728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828763.871315:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828763.871321:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828763.871326:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828763.871333:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3703 00000100:00080000:2.0:1787828763.871340:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828763.871346:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828763.871357:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828763.871379:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564313856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828763.871388:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828763.871409:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828763.871416:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828763.871420:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828763.871430:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3703 00000100:00080000:2.0:1787828763.871438:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828763.871446:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828763.871457:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828763.871459:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828763/real 1787828763] req@ffff976889cfbc00 x1874670564313600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828763.871490:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564313984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828763.871501:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564313600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828763.871533:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828763.871540:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828763.871543:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828763.871552:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828763.871555:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:2.0:1787828763.871568:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3703 00000100:00100000:1.0:1787828763.871571:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564313600:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828763.871582:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828763.871595:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828763.871605:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828763.871617:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564313728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828763.871633:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564314112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828763.871670:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828763.871704:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828763/real 1787828763] req@ffff976889cfa580 x1874670564313728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828763.871730:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564313728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828763.871747:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828763.871754:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828763.871762:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564313728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828763.871774:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564313856:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828763.871812:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828763.871900:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828763/real 1787828763] req@ffff976889cfb0c0 x1874670564313856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828763.871926:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564313856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828763.871970:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828763.871979:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828763.871987:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564313856:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828763.872001:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564313984:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828763.872044:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828763.872085:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828763/real 1787828763] req@ffff976889cf8000 x1874670564313984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828763.872109:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564313984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828763.872125:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828763.872131:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828763.872138:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564313984:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828763.872170:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564314112:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828763.872209:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828763.872232:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828763/real 1787828763] req@ffff976889cf8f00 x1874670564314112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828763.872254:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564314112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828763.872269:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828763.872276:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828763.872282:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564314112:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828764.127064:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828764.127076:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828764.127112:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828764.894905:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828765.150886:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828765.150942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828765.150959:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828765.918941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828766.174957:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828766.174959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828766.174983:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828766.942937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828767.198897:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828767.198922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828767.198967:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828767.966937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828768.222970:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828768.222979:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828768.223013:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828768.990978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828768.990985:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828768.990988:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828768.990994:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828768.991005:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3708 00000100:00080000:2.0:1787828768.991019:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828768.991027:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828768.991072:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564314240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828768.991099:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828768.991102:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828768.991104:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828768.991107:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3708 00000100:00080000:2.0:1787828768.991110:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828768.991112:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828768.991122:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828768.991131:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564314368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828768.991147:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828768.991154:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787828768.991155:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564314240:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828768.991156:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828768.991170:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3708 00000100:00080000:2.0:1787828768.991176:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828768.991179:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828768.991182:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828768.991193:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564314496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828768.991199:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828768.991202:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828768.991204:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828768.991207:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3708 00000100:00080000:2.0:1787828768.991210:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828768.991212:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828768.991216:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828768.991224:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564314624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828768.991230:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828768.991232:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828768.991233:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828768.991237:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3708 00000100:00080000:2.0:1787828768.991240:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828768.991243:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828768.991246:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828768.991253:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828768.991257:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564314752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828768.991312:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828768/real 1787828768] req@ffff976889cfa940 x1874670564314240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828768.991334:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564314240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828768.991348:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828768.991353:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828768.991358:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564314240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828768.991385:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564314368:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828768.991414:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828768.991472:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828768/real 1787828768] req@ffff976889cfad00 x1874670564314368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828768.991491:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564314368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828768.991501:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828768.991506:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828768.991509:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564314368:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828768.991517:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564314496:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828768.991542:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828768.991556:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828768/real 1787828768] req@ffff976889cf92c0 x1874670564314496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828768.991570:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564314496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828768.991580:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828768.991584:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828768.991588:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564314496:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828768.991597:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564314624:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828768.991621:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828768.991634:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828768/real 1787828768] req@ffff976889cf9e00 x1874670564314624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828768.991648:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564314624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828768.991658:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828768.991662:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828768.991666:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564314624:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828768.991673:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564314752:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828768.991696:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828768.991709:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828768/real 1787828768] req@ffff976889cfb480 x1874670564314752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828768.991723:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564314752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828768.991733:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828768.991737:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828768.991741:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564314752:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828769.246932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828769.246988:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828769.247074:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828770.014958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828770.271035:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828770.271051:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828770.271097:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828771.038905:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828771.294964:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828771.294971:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828771.295052:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828772.062949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828772.318956:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828772.318981:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828772.319063:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828773.086887:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828773.342922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828773.342956:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828773.343001:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828774.110954:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828774.110959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828774.110966:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828774.110976:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828774.110989:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3714 00000100:00080000:2.0:1787828774.111004:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828774.111013:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828774.111058:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564314880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828774.111086:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828774.111107:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828774.111110:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828774.111115:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3714 00000100:00080000:2.0:1787828774.111120:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828774.111127:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564314880:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828774.111131:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828774.111168:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828774.111188:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564315008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828774.111203:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828774.111209:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828774.111212:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828774.111233:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3714 00000100:00080000:2.0:1787828774.111239:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828774.111245:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828774.111254:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828774.111279:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564315136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828774.111294:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828774.111297:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828774.111300:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828774.111304:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3714 00000100:00080000:2.0:1787828774.111308:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828774.111314:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828774.111321:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828774.111335:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564315264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828774.111345:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828774.111349:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828774.111351:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828774.111356:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3714 00000100:00080000:2.0:1787828774.111362:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828774.111364:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564315008:192.168.203.139@tcp:38:kworker.0 00000100:00080000:2.0:1787828774.111370:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828774.111383:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828774.111394:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564315392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828774.111539:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564315136:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828774.111594:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564315264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828774.111659:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564315392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828774.366934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828774.366934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828774.367031:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828775.135091:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828775.390934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828775.390989:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828775.390990:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828776.159039:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828776.414989:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828776.414989:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828776.415052:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787828777.183124:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000100:00100000:1.0:1787828777.183124:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:0.0:1787828777.183169:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828777.183187:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828777.183223:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828777.183235:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828777.183245:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828777.183254:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:3.0:1787828777.183265:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828774/real 1787828777] req@ffff976889cfb480 x1874670564315008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828777.183293:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564315008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828777.183306:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828777.183311:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828777.183315:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564315008:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828777.183323:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828774/real 1787828777] req@ffff976889cf9e00 x1874670564315136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828777.183335:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564315136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828777.183344:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828777.183348:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828777.183351:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564315136:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828777.183357:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828774/real 1787828777] req@ffff976889cf92c0 x1874670564315264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828777.183368:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564315264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828777.183375:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828777.183379:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828777.183382:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564315264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828777.183388:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828774/real 1787828777] req@ffff976889cfa1c0 x1874670564314880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828777.183399:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564314880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828777.183405:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828777.183408:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828777.183411:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564314880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828777.183437:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828774/real 1787828777] req@ffff976889cfad00 x1874670564315392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828777.183449:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564315392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828777.183457:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828777.183460:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828777.183464:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564315392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828777.438930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828777.438975:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828777.438989:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828778.207050:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828778.462944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828778.462964:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828778.463014:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828779.230964:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828779.230973:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828779.230979:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828779.230990:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3719 00000100:00080000:2.0:1787828779.231004:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828779.231008:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828779.231013:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828779.231066:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564315520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828779.231101:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828779.231105:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828779.231108:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828779.231113:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3719 00000100:00080000:2.0:1787828779.231119:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828779.231125:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828779.231140:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828779.231151:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564315520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828779.231158:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564315648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828779.231171:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828779.231175:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828779.231179:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828779.231185:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3719 00000100:00080000:2.0:1787828779.231189:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828779.231195:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828779.231202:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828779.231220:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564315776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828779.231231:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828779.231236:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828779.231238:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787828779.231255:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828779.231270:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3719 00000100:00080000:2.0:1787828779.231277:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828779.231283:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828779.231291:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828779.231313:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564315904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828779.231336:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828779/real 1787828779] req@ffff976889cfa940 x1874670564315520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828779.231348:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828779.231363:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828779.231367:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828779.231380:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3719 00000100:00080000:2.0:1787828779.231387:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828779.231397:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828779.231399:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564315520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828779.231415:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787828779.231421:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828779.231433:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828779.231438:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564316032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828779.231446:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564315520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828779.231506:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564315648:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828779.231536:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828779.231550:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828779/real 1787828779] req@ffff976889cf8f00 x1874670564315648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828779.231564:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564315648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828779.231573:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828779.231577:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828779.231581:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564315648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828779.231589:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564315776:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828779.231611:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828779.231620:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828779/real 1787828779] req@ffff976889cf8000 x1874670564315776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828779.231633:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564315776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828779.231642:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828779.231647:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828779.231650:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564315776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828779.231659:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564315904:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828779.231682:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828779.231692:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828779/real 1787828779] req@ffff976889cfb0c0 x1874670564315904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828779.231714:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564315904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828779.231721:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828779.231725:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828779.231728:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564315904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828779.231735:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564316032:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828779.231873:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828779.231888:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828779/real 1787828779] req@ffff976889cfa580 x1874670564316032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828779.231902:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564316032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828779.231911:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828779.231914:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828779.231918:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564316032:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828779.486960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828779.486975:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828779.487020:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828780.255019:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828780.510915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828780.510926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828780.510945:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828781.278941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828781.534899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828781.534906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828781.534946:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828782.302935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828782.558906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828782.558913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828782.558991:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828783.326959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828783.582935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828783.582935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828783.583002:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828784.350940:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828784.350940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828784.350950:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828784.350955:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828784.350963:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3724 00000100:00080000:0.0:1787828784.350973:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828784.350979:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828784.351021:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564316160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828784.351048:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828784.351052:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828784.351054:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828784.351058:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3724 00000100:00080000:0.0:1787828784.351062:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828784.351067:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828784.351077:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828784.351086:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564316160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828784.351091:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564316288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828784.351101:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828784.351104:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828784.351106:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828784.351109:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3724 00000100:00080000:0.0:1787828784.351112:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828784.351115:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828784.351120:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828784.351129:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564316416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828784.351137:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828784.351141:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828784.351142:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787828784.351143:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828784.351146:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3724 00000100:00080000:0.0:1787828784.351149:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828784.351152:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828784.351156:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828784.351165:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564316544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828784.351173:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828784.351176:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828784.351178:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828784.351181:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3724 00000100:00100000:3.0:1787828784.351182:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828784/real 1787828784] req@ffff976882aed680 x1874670564316160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828784.351184:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828784.351188:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828784.351192:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828784.351200:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564316672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828784.351200:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564316160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828784.351211:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828784.351215:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828784.351218:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564316160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828784.351250:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564316288:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828784.351273:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828784.351283:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828784/real 1787828784] req@ffff976882aee940 x1874670564316288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828784.351291:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564316288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828784.351302:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828784.351304:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828784.351307:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564316288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828784.351311:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564316416:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828784.351322:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828784.351328:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828784/real 1787828784] req@ffff976882aef0c0 x1874670564316416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828784.351336:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564316416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828784.351341:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828784.351343:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828784.351345:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564316416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828784.351348:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564316544:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828784.351358:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828784.351364:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828784/real 1787828784] req@ffff976882aeed00 x1874670564316544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828784.351371:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564316544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828784.351376:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828784.351378:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828784.351380:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564316544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828784.351383:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564316672:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828784.351393:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828784.351398:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828784/real 1787828784] req@ffff976882aee580 x1874670564316672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828784.351405:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564316672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828784.351410:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828784.351411:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828784.351413:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564316672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828784.606904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828784.606904:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828784.607033:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828785.374941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828785.630948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828785.630968:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828785.631038:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828786.398934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828786.654916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828786.654916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828786.654970:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828787.422913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828787.678945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828787.678945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828787.679008:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828788.446893:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828788.702939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828788.702941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828788.703037:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828789.470946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828789.470949:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828789.470959:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828789.470964:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828789.470974:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3729 00000100:00080000:2.0:1787828789.470985:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828789.470992:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828789.471040:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564316800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828789.471069:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828789.471076:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828789.471078:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828789.471082:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3729 00000100:00080000:2.0:1787828789.471086:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828789.471091:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828789.471101:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828789.471117:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564316800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828789.471120:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564316928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828789.471137:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828789.471140:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828789.471142:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828789.471146:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3729 00000100:00080000:2.0:1787828789.471150:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828789.471162:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828789.471167:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828789.471179:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564317056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828789.471189:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828789.471192:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828789.471194:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787828789.471200:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828789.471202:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3729 00000100:00080000:2.0:1787828789.471208:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828789.471213:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828789.471220:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828789.471233:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564317184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828789.471243:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828789.471248:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828789/real 1787828789] req@ffff976889cfbc00 x1874670564316800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828789.471249:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828789.471252:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828789.471261:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3729 00000100:00080000:2.0:1787828789.471267:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828789.471274:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828789.471279:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828789.471281:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564316800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828789.471294:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564317312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828789.471296:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828789.471307:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828789.471314:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564316800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828789.471338:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564316928:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828789.471362:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828789.471375:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828789/real 1787828789] req@ffff976889cfa580 x1874670564316928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828789.471389:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564316928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828789.471398:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828789.471401:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828789.471404:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564316928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828789.471411:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564317056:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828789.471432:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828789.471443:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828789/real 1787828789] req@ffff976889cfb0c0 x1874670564317056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828789.471456:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564317056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828789.471464:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828789.471467:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828789.471480:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564317056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828789.471487:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564317184:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828789.471507:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828789.471519:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828789/real 1787828789] req@ffff976889cf8000 x1874670564317184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828789.471532:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564317184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828789.471541:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828789.471544:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828789.471548:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564317184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828789.471554:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564317312:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828789.471572:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828789.471583:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828789/real 1787828789] req@ffff976889cf8f00 x1874670564317312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828789.471595:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564317312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828789.471603:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828789.471607:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828789.471610:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564317312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828789.726940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828789.726942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828789.727017:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828790.494901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828790.750914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828790.750915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828790.750949:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828791.518938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828791.774977:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828791.774986:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828791.775059:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828792.542960:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828792.798948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828792.798998:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828792.799017:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828793.566923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828793.822935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828793.822935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828793.823004:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828794.590930:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828794.590935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828794.590948:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828794.590954:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828794.590966:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3734 00000100:00080000:0.0:1787828794.590979:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828794.590987:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828794.591028:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564317440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828794.591060:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828794.591064:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828794.591066:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828794.591072:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3734 00000100:00080000:0.0:1787828794.591078:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828794.591083:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828794.591094:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828794.591104:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564317440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828794.591113:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564317568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828794.591129:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828794.591133:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828794.591135:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828794.591139:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3734 00000100:00080000:0.0:1787828794.591143:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828794.591147:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828794.591154:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828794.591168:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564317696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828794.591179:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828794.591183:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828794.591185:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828794.591189:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3734 00000100:00080000:0.0:1787828794.591193:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787828794.591199:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828794.591200:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828794.591209:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828794.591224:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564317824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828794.591235:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828794.591240:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828794.591242:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828794.591247:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3734 00000100:00080000:0.0:1787828794.591251:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828794.591258:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828794/real 1787828794] req@ffff976882aec3c0 x1874670564317440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828794.591259:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828794.591275:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828794.591304:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564317952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828794.591307:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564317440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828794.591380:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828794.591385:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828794.591390:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564317440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828794.591419:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564317568:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828794.591441:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828794.591451:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828794/real 1787828794] req@ffff976882aee580 x1874670564317568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828794.591460:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564317568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828794.591467:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828794.591469:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828794.591472:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564317568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828794.591476:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564317696:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828794.591488:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828794.591494:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828794/real 1787828794] req@ffff976882aeed00 x1874670564317696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828794.591502:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564317696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828794.591508:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828794.591510:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828794.591511:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564317696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828794.591516:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564317824:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828794.591528:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828794.591560:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828794/real 1787828794] req@ffff976882aef0c0 x1874670564317824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828794.591569:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564317824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828794.591575:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828794.591577:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828794.591579:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564317824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828794.591582:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564317952:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828794.591615:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828794.591622:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828794/real 1787828794] req@ffff976882aee940 x1874670564317952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828794.591630:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564317952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828794.591635:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828794.591637:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828794.591639:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564317952:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828794.846931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828794.846943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828794.846992:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828795.614985:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828795.870938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828795.870981:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828795.871150:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828796.638983:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828796.894897:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828796.894952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828796.894969:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828797.662918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828797.918955:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828797.918965:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828797.919031:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828798.686951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828798.942942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828798.943021:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828798.943032:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828799.710886:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828799.710945:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828799.710953:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828799.710958:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828799.710968:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3739 00000100:00080000:0.0:1787828799.710982:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828799.710989:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828799.711032:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564318080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828799.711056:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828799.711059:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828799.711061:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828799.711064:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3739 00000100:00080000:0.0:1787828799.711066:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828799.711070:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828799.711081:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828799.711093:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564318208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828799.711102:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564318080:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828799.711115:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828799.711119:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828799.711120:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828799.711123:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3739 00000100:00080000:0.0:1787828799.711125:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828799.711130:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828799.711134:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828799.711150:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564318336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828799.711167:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828799.711170:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828799.711172:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828799.711175:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3739 00000100:00080000:0.0:1787828799.711179:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828799.711182:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828799.711186:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828799.711194:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564318464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828799.711200:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828799.711210:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828799.711211:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828799.711217:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828799.711222:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3739 00000100:00080000:0.0:1787828799.711227:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828799.711230:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828799.711233:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828799.711245:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564318592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828799.711288:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828799/real 1787828799] req@ffff976882aed680 x1874670564318080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828799.711304:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564318080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828799.711320:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828799.711323:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828799.711326:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564318080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828799.711348:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564318208:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828799.711368:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828799.711377:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828799/real 1787828799] req@ffff976882aee940 x1874670564318208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828799.711386:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564318208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828799.711391:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828799.711394:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828799.711396:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564318208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828799.711400:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564318336:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828799.711411:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828799.711417:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828799/real 1787828799] req@ffff976882aef0c0 x1874670564318336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828799.711424:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564318336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828799.711429:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828799.711431:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828799.711433:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564318336:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828799.711437:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564318464:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828799.711448:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828799.711454:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828799/real 1787828799] req@ffff976882aeed00 x1874670564318464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828799.711461:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564318464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828799.711467:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828799.711469:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828799.711471:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564318464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828799.711474:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564318592:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828799.711484:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828799.711489:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828799/real 1787828799] req@ffff976882aee580 x1874670564318592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828799.711496:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564318592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828799.711500:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828799.711502:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828799.711504:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564318592:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828799.966922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828799.966922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828799.967021:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828800.734955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828800.990980:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828800.991004:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828800.991044:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828801.758962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828802.014949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828802.014956:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828802.015035:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828802.782939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828803.038904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828803.038953:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828803.039014:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828803.806937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828804.062944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828804.062945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828804.063023:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828804.830930:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828804.830941:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1787828804.830946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828804.830948:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828804.830965:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3744 00000100:00080000:1.0:1787828804.830981:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828804.830992:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828804.831052:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564318720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828804.831088:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828804.831092:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828804.831095:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828804.831100:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3744 00000100:00080000:1.0:1787828804.831104:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828804.831110:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828804.831123:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828804.831138:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564318720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828804.831145:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564318848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828804.831166:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828804.831171:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828804.831174:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828804.831180:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3744 00000100:00080000:1.0:1787828804.831185:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828804.831190:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828804.831196:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828804.831211:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564318976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828804.831222:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787828804.831227:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828804.831233:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828804.831236:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828804.831243:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3744 00000100:00080000:1.0:1787828804.831257:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828804.831262:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828804.831269:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828804.831274:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828804/real 1787828804] req@ffff976889efc3c0 x1874670564318720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828804.831288:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564319104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828804.831296:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564318720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828804.831320:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828804.831322:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828804.831329:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828804.831330:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:1.0:1787828804.831336:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828804.831337:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564318720:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787828804.831349:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3744 00000100:00080000:1.0:1787828804.831355:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828804.831361:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828804.831367:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828804.831381:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564319232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828804.831387:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564318848:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828804.831429:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828804.831447:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828804/real 1787828804] req@ffff976889efed00 x1874670564318848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828804.831464:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564318848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828804.831486:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828804.831490:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828804.831495:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564318848:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828804.831501:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564318976:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828804.831520:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828804.831530:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828804/real 1787828804] req@ffff976889effc00 x1874670564318976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828804.831539:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564318976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828804.831551:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828804.831554:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828804.831557:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564318976:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828804.831560:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564319104:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828804.831573:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828804.831580:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828804/real 1787828804] req@ffff976889eff0c0 x1874670564319104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828804.831589:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564319104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828804.831595:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828804.831598:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828804.831600:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564319104:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828804.831617:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564319232:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828804.831629:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828804.831636:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828804/real 1787828804] req@ffff976889efe940 x1874670564319232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828804.831644:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564319232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828804.831667:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828804.831670:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828804.831673:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564319232:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828805.086922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828805.086933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828805.087032:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828805.855108:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828806.110961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828806.110973:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828806.111015:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828806.878928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828807.134939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828807.134939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828807.135008:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828807.902909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828808.158910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828808.158918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787828808.158962:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828808.926973:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828809.182982:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828809.182983:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828809.183082:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828809.950955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828809.951076:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828809.951097:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828809.951120:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828809.951132:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3749 00000100:00080000:0.0:1787828809.951144:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828809.951165:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828809.951215:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564319360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828809.951249:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828809.951255:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828809.951258:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828809.951265:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3749 00000100:00080000:0.0:1787828809.951274:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828809.951285:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828809.951303:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564319360:192.168.203.139@tcp:250:kworker.0 00000008:00100000:0.0:1787828809.951312:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828809.951340:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564319488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828809.951353:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828809.951357:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828809.951359:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828809.951366:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3749 00000100:00080000:0.0:1787828809.951371:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828809.951378:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:1.0:1787828809.951380:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:0.0:1787828809.951385:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828809.951407:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564319616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828809.951417:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828809.951421:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828809.951423:0:36:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828809.951428:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3749 00000100:00080000:0.0:1787828809.951432:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828809.951434:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828809/real 1787828809] req@ffff976882aec3c0 x1874670564319360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828809.951441:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828809.951455:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828809.951476:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564319744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828809.951483:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564319360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828809.951493:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828809.951514:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828809.951519:0:36:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828809.951529:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3749 00000100:00080000:1.0:1787828809.951530:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828809.951533:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828809.951543:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:1.0:1787828809.951547:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000008:00100000:0.0:1787828809.951552:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828809.951558:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564319360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828809.951572:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564319872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828809.951594:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564319488:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828809.951650:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828809.951686:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828809/real 1787828809] req@ffff976882aee580 x1874670564319488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828809.951700:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564319488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828809.951711:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828809.951714:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828809.951718:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564319488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828809.951725:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564319616:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828809.951746:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828809.951757:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828809/real 1787828809] req@ffff976882aeed00 x1874670564319616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828809.951780:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564319616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828809.951788:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828809.951791:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828809.951794:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564319616:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828809.951800:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564319744:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828809.951817:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828809.951826:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828809/real 1787828809] req@ffff976882aef0c0 x1874670564319744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828809.951875:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564319744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828809.951885:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828809.951889:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828809.951893:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564319744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828809.951901:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564319872:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828809.951944:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828809.951954:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828809/real 1787828809] req@ffff976882aee940 x1874670564319872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828809.951965:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564319872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828809.951974:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828809.951976:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828809.951997:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564319872:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828810.206922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828810.206969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828810.207144:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828810.974928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828811.230978:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828811.230978:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828811.231122:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828811.998898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828812.254948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828812.254948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828812.255034:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828813.022920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828813.278941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828813.278948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828813.279019:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828814.046955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828814.302965:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828814.302965:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828814.303026:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828815.070967:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828815.070971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828815.070982:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828815.070987:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828815.070998:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3754 00000100:00080000:3.0:1787828815.071009:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828815.071018:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828815.071056:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564320000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828815.071085:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828815.071089:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828815.071091:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828815.071095:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3754 00000100:00080000:3.0:1787828815.071099:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828815.071103:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828815.071113:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828815.071132:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564320000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828815.071132:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564320128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828815.071149:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828815.071153:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828815.071155:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828815.071159:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3754 00000100:00080000:3.0:1787828815.071162:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828815.071167:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828815.071173:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828815.071185:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564320256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828815.071196:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787828815.071199:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828815.071213:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828815.071216:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828815.071221:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3754 00000100:00080000:3.0:1787828815.071227:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828815.071233:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828815/real 1787828815] req@ffff97688cb5d2c0 x1874670564320000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828815.071234:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828815.071245:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828815.071254:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564320000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828815.071264:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828815.071264:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564320384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828815.071272:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828815.071279:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564320000:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787828815.071290:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828815.071295:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828815.071297:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828815.071302:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3754 00000100:00100000:1.0:1787828815.071306:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564320128:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787828815.071307:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828815.071315:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828815.071319:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828815.071330:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828815.071334:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564320512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828815.071345:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828815/real 1787828815] req@ffff97688cb5f0c0 x1874670564320128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828815.071361:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564320128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828815.071367:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828815.071370:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828815.071372:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564320128:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828815.071377:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564320256:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828815.071388:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828815.071395:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828815/real 1787828815] req@ffff97688cb5c3c0 x1874670564320256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828815.071402:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564320256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828815.071407:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828815.071409:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828815.071411:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564320256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828815.071415:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564320384:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828815.071425:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828815.071431:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828815/real 1787828815] req@ffff97688cb5c780 x1874670564320384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828815.071438:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564320384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828815.071448:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828815.071450:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828815.071452:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564320384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828815.071464:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564320512:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828815.071474:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828815.071480:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828815/real 1787828815] req@ffff97688cb5e1c0 x1874670564320512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828815.071487:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564320512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828815.071492:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828815.071494:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828815.071496:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564320512:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828815.326932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828815.326932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828815.327008:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828816.094968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828816.350907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828816.350929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828816.350973:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828817.118976:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828817.374957:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828817.374958:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828817.375070:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828818.142969:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828818.398942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828818.398942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828818.399027:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828819.166912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828819.422940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828819.422948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828819.423021:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828820.190951:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828820.190955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828820.190965:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828820.190970:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828820.190981:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3760 00000100:00080000:0.0:1787828820.190993:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828820.190999:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828820.191040:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564320640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828820.191064:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828820.191067:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828820.191069:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828820.191071:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3760 00000100:00080000:0.0:1787828820.191075:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828820.191078:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828820.191086:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828820.191099:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564320768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828820.191106:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564320640:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828820.191110:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828820.191114:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828820.191115:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828820.191118:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3760 00000100:00080000:0.0:1787828820.191120:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828820.191123:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828820.191127:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828820.191141:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564320896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828820.191151:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828820.191155:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828820.191173:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828820.191181:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828820.191182:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3760 00000100:00080000:0.0:1787828820.191190:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828820.191197:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828820.191204:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828820.191219:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564321024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828820.191239:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828820/real 1787828820] req@ffff976882aed680 x1874670564320640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828820.191261:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828820.191267:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828820.191269:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828820.191275:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3760 00000100:00100000:1.0:1787828820.191276:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564320640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828820.191280:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828820.191288:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:1.0:1787828820.191292:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828820.191294:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828820.191297:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828820.191302:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564320640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828820.191309:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564321152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828820.191322:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564320768:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828820.191341:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828820.191350:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828820/real 1787828820] req@ffff976882aee940 x1874670564320768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828820.191358:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564320768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828820.191364:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828820.191366:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828820.191368:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564320768:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828820.191371:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564320896:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828820.191381:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828820.191387:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828820/real 1787828820] req@ffff976882aef0c0 x1874670564320896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828820.191410:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564320896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828820.191416:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828820.191418:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828820.191420:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564320896:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828820.191425:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564321024:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828820.191438:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828820.191445:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828820/real 1787828820] req@ffff976882aeed00 x1874670564321024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828820.191453:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564321024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828820.191459:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828820.191461:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828820.191463:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564321024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828820.191466:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564321152:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828820.191476:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828820.191482:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828820/real 1787828820] req@ffff976882aee580 x1874670564321152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828820.191489:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564321152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828820.191495:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828820.191497:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828820.191499:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564321152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828820.446944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828820.446944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828820.447013:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828821.214929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828821.470971:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828821.470998:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828821.471022:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828822.238934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828822.494906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828822.494906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828822.494975:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828823.262898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828823.518896:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828823.518947:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828823.519075:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828824.286950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828824.542895:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828824.542907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828824.542958:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828825.310892:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828825.310896:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828825.310904:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828825.310908:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828825.310916:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3765 00000100:00080000:0.0:1787828825.310926:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828825.310932:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828825.310963:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564321280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828825.310986:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828825.310989:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828825.310990:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828825.310993:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3765 00000100:00080000:0.0:1787828825.310995:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828825.310997:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828825.311005:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828825.311015:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564321408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828825.311023:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828825.311026:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828825.311027:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828825.311030:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3765 00000100:00080000:0.0:1787828825.311031:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828825.311036:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828825.311038:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564321280:192.168.203.139@tcp:250:kworker.0 00000008:00100000:0.0:1787828825.311039:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828825.311049:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564321536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828825.311057:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828825.311058:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828825.311059:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828825.311062:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3765 00000100:00080000:0.0:1787828825.311064:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828825.311066:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828825.311070:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828825.311078:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564321664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828825.311085:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828825.311087:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828825.311088:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828825.311090:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3765 00000100:00080000:0.0:1787828825.311095:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828825.311097:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828825.311099:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828825.311109:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564321792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828825.311109:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828825.311155:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828825/real 1787828825] req@ffff976882aec3c0 x1874670564321280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828825.311169:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564321280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828825.311177:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828825.311180:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828825.311183:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564321280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828825.311188:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564321408:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828825.311210:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828825.311223:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828825/real 1787828825] req@ffff976882aee580 x1874670564321408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828825.311236:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564321408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828825.311246:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828825.311250:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828825.311252:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564321408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828825.311272:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564321536:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828825.311287:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828825.311295:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828825/real 1787828825] req@ffff976882aeed00 x1874670564321536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828825.311303:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564321536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828825.311309:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828825.311311:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828825.311313:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564321536:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828825.311316:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564321664:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828825.311326:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828825.311332:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828825/real 1787828825] req@ffff976882aef0c0 x1874670564321664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828825.311340:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564321664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828825.311345:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828825.311346:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828825.311348:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564321664:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828825.311351:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564321792:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828825.311360:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828825.311365:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828825/real 1787828825] req@ffff976882aee940 x1874670564321792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828825.311372:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564321792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828825.311382:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828825.311384:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828825.311386:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564321792:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828825.566889:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828825.566889:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828825.566939:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828826.334944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828826.590901:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828826.590906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828826.590911:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828827.358896:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828827.614904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828827.614961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828827.614987:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828828.382953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828828.638944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828828.638944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828828.638978:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828829.406906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828829.662917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828829.662917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828829.662988:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828830.430911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828830.430921:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828830.430928:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828830.430933:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828830.430943:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3770 00000100:00080000:0.0:1787828830.430953:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828830.430961:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828830.430998:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564321920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828830.431029:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828830.431033:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828830.431035:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828830.431039:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3770 00000100:00080000:0.0:1787828830.431043:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828830.431048:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828830.431058:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828830.431070:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564321920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828830.431075:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564322048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828830.431087:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828830.431091:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828830.431093:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828830.431105:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3770 00000100:00080000:0.0:1787828830.431118:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828830.431123:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:1.0:1787828830.431127:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:0.0:1787828830.431131:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828830.431149:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564322176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828830.431160:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828830.431163:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787828830.431167:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828830/real 1787828830] req@ffff976882aed680 x1874670564321920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828830.431168:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828830.431179:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3770 00000100:00080000:0.0:1787828830.431186:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828830.431191:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564321920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828830.431193:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828830.431202:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1787828830.431207:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828830.431214:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828830.431219:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564321920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828830.431228:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564322304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828830.431240:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828830.431243:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787828830.431247:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564322048:192.168.203.139@tcp:38:kworker.0 00000100:00080000:0.0:1787828830.431248:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828830.431257:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3770 00000100:00080000:0.0:1787828830.431261:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828830.431266:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828830.431272:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828830.431285:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828830.431290:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564322432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828830.431305:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828830/real 1787828830] req@ffff976882aee940 x1874670564322048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828830.431320:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564322048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828830.431329:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828830.431333:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828830.431336:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564322048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828830.431342:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564322176:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828830.431361:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828830.431370:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828830/real 1787828830] req@ffff976882aef0c0 x1874670564322176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828830.431381:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564322176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828830.431389:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828830.431393:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828830.431396:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564322176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828830.431402:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564322304:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828830.431421:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828830.431433:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828830/real 1787828830] req@ffff976882aeed00 x1874670564322304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828830.431444:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564322304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828830.431450:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828830.431452:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828830.431454:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564322304:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828830.431480:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564322432:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828830.431496:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828830.431504:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828830/real 1787828830] req@ffff976882aee580 x1874670564322432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828830.431520:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564322432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828830.431525:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828830.431527:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828830.431529:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564322432:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828830.686939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828830.686940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828830.686994:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828831.454970:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828831.710961:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828831.710961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828831.711001:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828832.478972:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828832.734922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828832.734922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828832.734979:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828833.502923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828833.758902:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828833.758902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828833.758989:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828834.526904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828834.782973:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828834.782973:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828834.783047:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828835.550964:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828835.551026:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828835.551034:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828835.551038:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828835.551063:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3775 00000100:00080000:3.0:1787828835.551073:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828835.551078:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828835.551143:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564322560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828835.551169:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828835.551172:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828835.551174:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828835.551177:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3775 00000100:00080000:3.0:1787828835.551180:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828835.551183:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828835.551192:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828835.551220:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564322688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828835.551230:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828835.551235:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564322560:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787828835.551236:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828835.551240:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828835.551260:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3775 00000100:00080000:3.0:1787828835.551264:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828835.551269:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828835.551275:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828835.551291:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564322816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828835.551303:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828835.551307:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828835.551309:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828835.551312:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787828835.551319:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3775 00000100:00080000:3.0:1787828835.551326:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828835.551332:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828835.551339:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828835.551352:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564322944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828835.551364:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828835/real 1787828835] req@ffff97688cb5e1c0 x1874670564322560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828835.551365:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828835.551376:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828835.551381:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828835.551389:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3775 00000100:00100000:1.0:1787828835.551392:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564322560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828835.551396:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828835.551407:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:1.0:1787828835.551409:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828835.551415:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000008:00100000:3.0:1787828835.551415:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828835.551422:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564322560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828835.551449:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564323072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828835.551478:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564322688:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828835.551502:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828835.551514:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828835/real 1787828835] req@ffff97688cb5c780 x1874670564322688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828835.551524:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564322688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828835.551530:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828835.551533:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828835.551536:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564322688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828835.551540:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564322816:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828835.551552:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828835.551559:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828835/real 1787828835] req@ffff97688cb5c3c0 x1874670564322816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828835.551567:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564322816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828835.551574:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828835.551577:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828835.551581:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564322816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828835.551588:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564322944:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828835.551611:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828835.551625:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828835/real 1787828835] req@ffff97688cb5f0c0 x1874670564322944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828835.551639:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564322944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828835.551646:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828835.551648:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828835.551650:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564322944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828835.551655:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564323072:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828835.551672:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828835.551680:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828835/real 1787828835] req@ffff97688cb5d2c0 x1874670564323072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828835.551688:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564323072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828835.551693:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828835.551695:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828835.551697:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564323072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828835.806914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828835.806923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828835.807006:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828836.574896:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828836.830927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828836.830927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828836.830954:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828837.598912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828837.854941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828837.854941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828837.855011:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828838.622900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828838.878900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828838.878907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828838.878992:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828839.646968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828839.902941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828839.902941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828839.902994:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828840.670933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828840.670933:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828840.670943:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828840.670947:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828840.670955:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3780 00000100:00080000:3.0:1787828840.670965:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828840.670982:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828840.671013:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564323200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828840.671036:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828840.671049:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828840.671050:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828840.671057:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3780 00000100:00080000:3.0:1787828840.671060:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828840.671066:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828840.671074:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828840.671085:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564323328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828840.671087:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564323200:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787828840.671100:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828840.671103:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828840.671105:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828840.671108:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3780 00000100:00080000:3.0:1787828840.671110:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828840.671112:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828840.671115:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828840.671123:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564323456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828840.671134:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828840.671139:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828840.671141:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828840.671146:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3780 00000100:00080000:3.0:1787828840.671150:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828840.671155:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828840.671176:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828840.671189:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564323584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828840.671200:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828840.671204:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828840.671207:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828840.671212:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3780 00000100:00080000:3.0:1787828840.671216:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828840.671220:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564323328:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787828840.671225:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828840.671231:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828840.671260:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564323712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828840.671276:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564323456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828840.671295:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564323584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828840.671342:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564323712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828840.926894:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828840.926902:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828840.926971:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828841.694936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828841.950947:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828841.950952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828841.951012:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828842.718946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828842.974925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828842.974938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828842.975060:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828843.742912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:1.0:1787828843.743128:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:1.0:1787828843.743169:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787828843.743182:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787828843.743211:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787828843.743232:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787828843.743244:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787828843.743252:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:3.0:1787828843.743262:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828840/real 1787828843] req@ffff97688cb5c000 x1874670564323328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828843.743301:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564323328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828843.743314:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828843.743318:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828843.743324:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564323328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828843.743332:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828840/real 1787828843] req@ffff97688cb5cb40 x1874670564323456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828843.743346:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564323456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828843.743354:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828843.743357:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828843.743361:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564323456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828843.743368:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828840/real 1787828843] req@ffff97688cb5de00 x1874670564323584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828843.743396:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564323584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828843.743405:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828843.743408:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828843.743412:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564323584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828843.743419:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828840/real 1787828843] req@ffff97688cb5d680 x1874670564323200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828843.743431:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564323200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828843.743438:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828843.743441:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828843.743444:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564323200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828843.743476:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828840/real 1787828843] req@ffff97688cb5f840 x1874670564323712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828843.743489:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564323712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828843.743497:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828843.743501:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828843.743504:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564323712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828843.998935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828843.998936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828843.998994:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828844.766934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828845.022929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828845.022972:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828845.023033:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828845.791016:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828845.791101:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828845.791106:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828845.791110:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828845.791118:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3785 00000100:00080000:2.0:1787828845.791139:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828845.791146:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828845.791179:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564323840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828845.791212:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828845.791215:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828845.791216:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828845.791221:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3785 00000100:00080000:2.0:1787828845.791226:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828845.791232:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828845.791244:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828845.791271:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564323968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828845.791283:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828845.791287:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828845.791289:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828845.791295:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3785 00000100:00080000:2.0:1787828845.791300:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828845.791305:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828845.791322:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828845.791338:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564324096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828845.791350:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828845.791354:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828845.791356:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828845.791375:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3785 00000100:00080000:2.0:1787828845.791393:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828845.791400:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828845.791411:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828845.791443:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564324224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828845.791459:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828845.791461:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828845.791463:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828845.791468:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3785 00000100:00080000:2.0:1787828845.791472:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828845.791478:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564323840:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828845.791480:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828845.791494:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828845.791512:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564324352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828845.791577:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828845.791632:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828845/real 1787828845] req@ffff976889cfa940 x1874670564323840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828845.791652:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564323840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828845.791665:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828845.791670:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828845.791675:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564323840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828845.791683:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564323968:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828845.791710:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828845.791723:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828845/real 1787828845] req@ffff976889cfad00 x1874670564323968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828845.791735:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564323968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828845.791744:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828845.791747:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828845.791750:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564323968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828845.791756:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564324096:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828845.791775:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828845.791786:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828845/real 1787828845] req@ffff976889cfa1c0 x1874670564324096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828845.791798:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564324096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828845.791806:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828845.791809:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828845.791813:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564324096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828845.791818:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564324224:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828845.791900:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828845.791913:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828845/real 1787828845] req@ffff976889cf92c0 x1874670564324224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828845.791925:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564324224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828845.791933:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828845.791938:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828845.791941:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564324224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828845.791968:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564324352:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828845.791988:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828845.792000:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828845/real 1787828845] req@ffff976889cf9e00 x1874670564324352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828845.792011:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564324352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828845.792019:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828845.792021:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828845.792024:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564324352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828846.046944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828846.046944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828846.046995:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828846.814903:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828847.070930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828847.070930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828847.070961:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828847.838943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828848.094940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828848.094941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828848.095004:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828848.862896:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828849.118942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828849.118942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828849.119026:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828849.886971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828850.142931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828850.142958:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828850.143009:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828850.910886:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828850.910897:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828850.910899:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828850.910906:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828850.910917:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3790 00000100:00080000:3.0:1787828850.910929:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828850.910935:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828850.910973:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564324480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828850.910994:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828850.910996:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828850.910998:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828850.911000:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3790 00000100:00080000:3.0:1787828850.911003:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828850.911008:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828850.911016:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828850.911025:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564324608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828850.911033:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828850.911035:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828850.911036:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828850.911040:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3790 00000100:00080000:3.0:1787828850.911041:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828850.911044:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828850.911047:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828850.911057:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564324480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828850.911057:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564324736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828850.911073:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828850.911076:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828850.911077:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828850.911079:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3790 00000100:00080000:3.0:1787828850.911081:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828850.911083:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828850.911086:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828850.911094:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564324864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828850.911100:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828850.911102:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828850.911104:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828850.911106:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3790 00000100:00080000:3.0:1787828850.911109:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828850.911112:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828850.911114:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828850.911121:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564324992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828850.911135:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828850.911189:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828850/real 1787828850] req@ffff97688cb5f840 x1874670564324480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828850.911219:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564324480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828850.911234:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828850.911238:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828850.911243:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564324480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828850.911251:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564324608:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828850.911275:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828850.911288:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828850/real 1787828850] req@ffff97688cb5d680 x1874670564324608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828850.911301:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564324608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828850.911310:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828850.911313:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828850.911317:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564324608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828850.911359:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564324736:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828850.911379:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828850.911391:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828850/real 1787828850] req@ffff97688cb5de00 x1874670564324736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828850.911404:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564324736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828850.911420:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828850.911423:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828850.911426:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564324736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828850.911432:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564324864:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828850.911450:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828850.911460:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828850/real 1787828850] req@ffff97688cb5cb40 x1874670564324864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828850.911472:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564324864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828850.911480:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828850.911484:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828850.911487:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564324864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828850.911493:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564324992:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828850.911511:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828850.911521:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828850/real 1787828850] req@ffff97688cb5c000 x1874670564324992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828850.911532:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564324992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828850.911541:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828850.911544:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828850.911548:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564324992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828851.166910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828851.166910:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828851.166959:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828851.934923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828852.190914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828852.190914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828852.190950:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828852.958961:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828853.214913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828853.214925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828853.214958:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828853.982895:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828854.238905:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828854.238906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828854.238939:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828855.006956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828855.262881:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828855.262926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828855.262995:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828856.030892:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828856.030902:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828856.030907:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828856.030908:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828856.030921:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3795 00000100:00080000:3.0:1787828856.030949:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828856.030960:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828856.031041:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564325120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828856.031070:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828856.031073:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828856.031074:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828856.031078:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3795 00000100:00080000:3.0:1787828856.031081:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828856.031085:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828856.031095:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828856.031107:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564325248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828856.031126:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828856.031129:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828856.031131:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828856.031133:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3795 00000100:00080000:3.0:1787828856.031136:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828856.031138:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828856.031141:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828856.031145:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564325120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828856.031173:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564325376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828856.031194:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828856.031197:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828856.031198:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828856.031202:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3795 00000100:00080000:3.0:1787828856.031205:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828856.031208:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828856.031212:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828856.031222:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564325504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828856.031229:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828856.031248:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828856.031260:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828856.031263:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3795 00000100:00080000:3.0:1787828856.031266:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828856.031270:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828856.031275:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828856.031288:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828856.031289:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564325632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828856.031382:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828856/real 1787828856] req@ffff97688cb5c000 x1874670564325120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828856.031397:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564325120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828856.031406:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828856.031409:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828856.031413:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564325120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828856.031419:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564325248:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828856.031438:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828856.031447:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828856/real 1787828856] req@ffff97688cb5cb40 x1874670564325248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828856.031455:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564325248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828856.031461:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828856.031464:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828856.031467:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564325248:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828856.031490:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564325376:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828856.031501:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828856.031508:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828856/real 1787828856] req@ffff97688cb5de00 x1874670564325376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828856.031516:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564325376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828856.031521:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828856.031523:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828856.031525:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564325376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828856.031528:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564325504:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828856.031540:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828856.031546:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828856/real 1787828856] req@ffff97688cb5d680 x1874670564325504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828856.031554:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564325504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828856.031559:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828856.031560:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828856.031562:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564325504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828856.031565:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564325632:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828856.031576:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828856.031582:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828856/real 1787828856] req@ffff97688cb5f840 x1874670564325632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828856.031590:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564325632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828856.031595:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828856.031597:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828856.031598:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564325632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828856.286978:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828856.287005:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828856.287028:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828857.054928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828857.310936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828857.310956:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828857.310966:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828858.078929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828858.334924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828858.334927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828858.334951:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828859.102979:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828859.358955:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828859.358959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828859.359030:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828860.126927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828860.382976:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828860.383021:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828860.383104:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828861.150954:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828861.150963:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828861.150967:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828861.150975:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3801 00000100:00080000:0.0:1787828861.150984:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828861.150993:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828861.151028:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564325760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828861.151053:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828861.151056:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828861.151058:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828861.151062:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3801 00000100:00080000:0.0:1787828861.151064:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828861.151068:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828861.151076:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000008:00100000:0.0:1787828861.151077:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828861.151096:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564325888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828861.151096:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564325760:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828861.151112:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828861.151115:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828861.151117:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828861.151120:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3801 00000100:00080000:0.0:1787828861.151123:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828861.151126:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828861.151130:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828861.151146:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564326016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828861.151153:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828861.151156:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828861.151157:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828861.151161:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3801 00000100:00080000:0.0:1787828861.151165:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828861.151168:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:3.0:1787828861.151172:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:0.0:1787828861.151173:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828861.151184:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564326144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828861.151192:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828861.151193:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828861.151195:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828861.151209:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3801 00000100:00080000:0.0:1787828861.151212:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828861.151216:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828861.151219:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828861.151223:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828861/real 1787828861] req@ffff976882aec3c0 x1874670564325760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828861.151230:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564326272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828861.151243:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564325760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828861.151252:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828861.151271:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828861.151274:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564325760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828861.151297:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564325888:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828861.151319:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828861.151330:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828861/real 1787828861] req@ffff976882aee580 x1874670564325888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828861.151338:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564325888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828861.151348:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828861.151350:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828861.151353:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564325888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828861.151356:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564326016:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828861.151367:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828861.151372:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828861/real 1787828861] req@ffff976882aeed00 x1874670564326016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828861.151379:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564326016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828861.151385:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828861.151386:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828861.151388:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564326016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828861.151392:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564326144:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828861.151401:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828861.151407:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828861/real 1787828861] req@ffff976882aef0c0 x1874670564326144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828861.151415:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564326144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828861.151420:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828861.151421:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828861.151424:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564326144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828861.151427:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564326272:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828861.151437:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828861.151442:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828861/real 1787828861] req@ffff976882aee940 x1874670564326272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828861.151449:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564326272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828861.151454:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828861.151455:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828861.151457:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564326272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828861.406919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828861.406919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828861.407001:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828862.174927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828862.430946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828862.430968:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828862.430975:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828863.198904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828863.454938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828863.454942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828863.455024:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828864.222908:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828864.478909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828864.478917:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828864.478943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828865.247055:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828865.502984:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828865.502985:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828865.503061:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828866.270889:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828866.270916:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828866.270920:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828866.270929:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3806 00000100:00080000:0.0:1787828866.270939:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828866.270941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828866.270947:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828866.270984:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564326400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828866.271010:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828866.271013:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828866.271014:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828866.271017:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3806 00000100:00080000:0.0:1787828866.271020:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828866.271022:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828866.271031:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828866.271042:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564326528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828866.271049:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828866.271052:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828866.271053:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828866.271056:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3806 00000100:00080000:0.0:1787828866.271058:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828866.271063:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828866.271065:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564326400:192.168.203.139@tcp:250:kworker.0 00000008:00100000:0.0:1787828866.271067:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828866.271079:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564326656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828866.271087:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828866.271105:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828866.271106:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828866.271110:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3806 00000100:00080000:0.0:1787828866.271112:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828866.271115:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828866.271119:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828866.271127:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564326784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828866.271133:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828866.271135:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828866.271137:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828866.271140:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3806 00000100:00080000:0.0:1787828866.271142:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828866.271144:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828866.271147:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787828866.271168:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828866.271170:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564326912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828866.271232:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828866/real 1787828866] req@ffff976882aed680 x1874670564326400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828866.271268:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564326400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828866.271280:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828866.271285:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828866.271289:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564326400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828866.271297:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564326528:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828866.271323:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828866.271338:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828866/real 1787828866] req@ffff976882aee940 x1874670564326528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828866.271352:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564326528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828866.271361:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828866.271365:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828866.271368:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564326528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828866.271393:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564326656:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828866.271414:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828866.271426:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828866/real 1787828866] req@ffff976882aef0c0 x1874670564326656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828866.271439:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564326656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828866.271448:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828866.271452:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828866.271455:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564326656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828866.271462:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564326784:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828866.271482:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828866.271509:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828866/real 1787828866] req@ffff976882aeed00 x1874670564326784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828866.271533:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564326784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828866.271544:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828866.271547:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828866.271551:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564326784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828866.271558:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564326912:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828866.271598:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828866.271613:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828866/real 1787828866] req@ffff976882aee580 x1874670564326912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828866.271627:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564326912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828866.271636:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828866.271640:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828866.271644:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564326912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828866.526926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828866.526926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828866.526952:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828867.294912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828867.550901:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828867.550959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828867.551016:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828868.318885:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828868.574917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828868.574946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828868.574987:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828869.342963:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828869.598978:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828869.598979:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828869.599051:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828870.366938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828870.622925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828870.622962:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828870.622992:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828871.391017:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828871.391040:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828871.391048:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828871.391053:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828871.391064:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3811 00000100:00080000:2.0:1787828871.391077:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828871.391084:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828871.391129:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564327040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828871.391157:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828871.391160:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828871.391162:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828871.391165:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3811 00000100:00080000:2.0:1787828871.391167:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828871.391170:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828871.391180:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828871.391206:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564327168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828871.391232:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564327040:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828871.391238:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828871.391244:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828871.391247:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828871.391253:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3811 00000100:00080000:2.0:1787828871.391257:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828871.391261:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828871.391267:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828871.391280:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564327296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828871.391291:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828871.391294:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828871.391296:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828871.391301:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3811 00000100:00080000:2.0:1787828871.391329:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828871.391332:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828871.391336:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787828871.391346:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828871.391347:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564327424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828871.391356:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828871.391359:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828871.391361:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828871.391365:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3811 00000100:00080000:2.0:1787828871.391367:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828871.391370:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828871.391391:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828871.391405:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564327552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828871.391428:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828871/real 1787828871] req@ffff976889cfb480 x1874670564327040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828871.391458:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564327040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828871.391469:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828871.391473:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828871.391477:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564327040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828871.391521:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564327168:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828871.391542:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828871.391583:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828871/real 1787828871] req@ffff976889cf9e00 x1874670564327168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828871.391596:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564327168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828871.391605:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828871.391609:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828871.391613:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564327168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828871.391649:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564327296:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828871.391680:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828871.391696:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828871/real 1787828871] req@ffff976889cf92c0 x1874670564327296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828871.391708:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564327296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828871.391749:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828871.391752:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828871.391762:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564327296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828871.391767:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564327424:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828871.391795:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828871.391803:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828871/real 1787828871] req@ffff976889cfa1c0 x1874670564327424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828871.391810:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564327424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828871.391816:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828871.391818:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828871.391820:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564327424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828871.391853:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564327552:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828871.391866:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828871.391873:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828871/real 1787828871] req@ffff976889cfad00 x1874670564327552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828871.391881:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564327552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828871.391887:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828871.391889:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828871.391892:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564327552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828871.646945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828871.646945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828871.647020:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828872.414938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828872.670891:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828872.670907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828872.670965:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828873.438914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828873.694956:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828873.694965:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828873.695056:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828874.462929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828874.718914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828874.718916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828874.719009:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828875.486949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828875.742926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828875.742976:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787828875.743001:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828876.510961:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828876.510972:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828876.510976:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828876.510986:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3816 00000100:00080000:0.0:1787828876.510996:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828876.511002:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828876.511039:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564327680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828876.511067:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828876.511070:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828876.511072:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828876.511075:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3816 00000100:00080000:0.0:1787828876.511078:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828876.511081:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828876.511091:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828876.511104:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564327808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828876.511113:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564327680:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787828876.511119:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828876.511124:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828876.511125:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828876.511129:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3816 00000100:00080000:0.0:1787828876.511131:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828876.511134:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828876.511139:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828876.511172:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564327936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828876.511182:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828876.511186:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828876.511187:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828876.511191:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3816 00000100:00080000:0.0:1787828876.511197:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787828876.511197:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787828876.511205:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828876.511211:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828876.511223:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564328064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828876.511231:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828876.511234:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828876.511236:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828876.511239:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3816 00000100:00080000:0.0:1787828876.511242:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828876.511245:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828876.511249:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828876.511258:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828876/real 1787828876] req@ffff976882aec3c0 x1874670564327680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828876.511261:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564328192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828876.511310:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564327680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828876.511313:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828876.511330:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828876.511337:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828876.511342:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564327680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828876.511379:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564327808:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828876.511409:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828876.511427:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828876/real 1787828876] req@ffff976882aee580 x1874670564327808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828876.511445:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564327808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828876.511453:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828876.511457:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828876.511460:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564327808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828876.511465:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564327936:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828876.511484:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828876.511494:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828876/real 1787828876] req@ffff976882aeed00 x1874670564327936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828876.511504:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564327936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828876.511511:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828876.511513:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828876.511516:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564327936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828876.511520:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564328064:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828876.511533:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828876.511540:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828876/real 1787828876] req@ffff976882aef0c0 x1874670564328064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828876.511549:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564328064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828876.511556:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828876.511558:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828876.511561:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564328064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828876.511565:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564328192:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828876.511577:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828876.511584:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828876/real 1787828876] req@ffff976882aee940 x1874670564328192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828876.511609:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564328192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828876.511615:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828876.511618:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828876.511620:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564328192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828876.766915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828876.766927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828876.766967:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828877.534904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828877.790960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828877.790970:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828877.791084:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828878.558904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828878.815001:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787828878.815024:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828878.815040:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828879.582950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828879.838909:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828879.838967:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828879.838970:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828880.606940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828880.862975:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828880.862978:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828880.863036:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828881.630955:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828881.630964:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1787828881.630967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828881.630969:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828881.630982:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3821 00000100:00080000:2.0:1787828881.630995:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828881.631002:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828881.631045:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564328320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828881.631075:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828881.631079:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828881.631080:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828881.631084:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3821 00000100:00080000:2.0:1787828881.631088:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828881.631092:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828881.631104:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828881.631135:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564328448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828881.631138:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564328320:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828881.631175:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828881.631179:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828881.631182:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828881.631187:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3821 00000100:00080000:2.0:1787828881.631194:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828881.631199:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828881.631205:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828881.631224:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564328576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828881.631228:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828881.631245:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828881.631249:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828881.631251:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828881.631256:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3821 00000100:00080000:2.0:1787828881.631259:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828881.631262:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828881.631268:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828881.631279:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564328704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828881.631290:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828881/real 1787828881] req@ffff976889cfa940 x1874670564328320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828881.631292:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828881.631299:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828881.631304:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828881.631314:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3821 00000100:00080000:2.0:1787828881.631325:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828881.631334:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828881.631338:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828881.631349:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564328320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828881.631354:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564328832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828881.631437:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828881.631446:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828881.631451:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564328320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828881.631480:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564328448:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828881.631530:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828881.631568:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828881/real 1787828881] req@ffff976889cf8f00 x1874670564328448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828881.631587:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564328448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828881.631598:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828881.631602:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828881.631606:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564328448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828881.631614:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564328576:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828881.631661:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828881.631687:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828881/real 1787828881] req@ffff976889cf8000 x1874670564328576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828881.631700:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564328576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828881.631707:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828881.631709:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828881.631712:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564328576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828881.631717:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564328704:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828881.631734:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828881.631742:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828881/real 1787828881] req@ffff976889cfb0c0 x1874670564328704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828881.631750:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564328704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828881.631756:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828881.631759:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828881.631761:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564328704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828881.631764:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564328832:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828881.631775:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828881.631782:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828881/real 1787828881] req@ffff976889cfa580 x1874670564328832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828881.631789:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564328832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828881.631795:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828881.631797:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828881.631800:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564328832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828881.886898:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828881.886979:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828881.886984:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828882.654939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828882.910966:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828882.910976:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828882.911015:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828883.678917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828883.934892:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828883.934902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828883.934973:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828884.702905:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828884.958904:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828884.958904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828884.958949:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828885.726917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828885.982931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828885.982931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828885.983004:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828886.750961:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828886.750963:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828886.750972:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828886.750977:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828886.750986:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3826 00000100:00080000:3.0:1787828886.750997:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828886.751005:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828886.751050:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564328960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828886.751092:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828886.751095:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828886.751097:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828886.751102:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3826 00000100:00080000:3.0:1787828886.751106:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828886.751111:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828886.751122:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828886.751130:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564328960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828886.751140:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564329088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828886.751150:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828886.751153:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828886.751154:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828886.751157:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3826 00000100:00080000:3.0:1787828886.751160:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828886.751162:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828886.751166:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828886.751174:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564329216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828886.751181:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828886.751184:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828886.751185:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828886.751189:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3826 00000100:00080000:3.0:1787828886.751192:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828886.751194:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828886.751198:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1787828886.751205:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828886.751209:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564329344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828886.751217:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828886.751220:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828886.751223:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828886.751225:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3826 00000100:00080000:3.0:1787828886.751228:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828886.751231:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828886.751233:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828886.751243:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564329472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828886.751254:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828886/real 1787828886] req@ffff97688cb5d2c0 x1874670564328960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828886.751274:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564328960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828886.751287:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828886.751292:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828886.751297:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564328960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828886.751329:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564329088:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828886.751359:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828886.751375:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828886/real 1787828886] req@ffff97688cb5f0c0 x1874670564329088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828886.751388:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564329088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828886.751396:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828886.751398:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828886.751401:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564329088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828886.751406:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564329216:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828886.751422:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828886.751430:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828886/real 1787828886] req@ffff97688cb5c3c0 x1874670564329216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828886.751438:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564329216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828886.751444:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828886.751446:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828886.751448:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564329216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828886.751451:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564329344:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828886.751463:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828886.751470:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828886/real 1787828886] req@ffff97688cb5c780 x1874670564329344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828886.751477:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564329344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828886.751482:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828886.751484:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828886.751487:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564329344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828886.751490:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564329472:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828886.751500:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828886.751506:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828886/real 1787828886] req@ffff97688cb5e1c0 x1874670564329472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828886.751513:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564329472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828886.751518:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828886.751520:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828886.751522:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564329472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828887.006926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828887.006931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828887.006933:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828887.774969:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828888.030995:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828888.031002:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828888.031007:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828888.798962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787828889.054976:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828889.054996:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787828889.055097:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828889.822997:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828890.078891:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828890.078905:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828890.078929:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828890.846931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828891.102930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828891.102930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828891.102993:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828891.870982:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828891.870994:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828891.870995:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828891.871004:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828891.871021:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3831 00000100:00080000:3.0:1787828891.871038:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828891.871070:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828891.871146:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564329600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828891.871185:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828891.871190:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828891.871192:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828891.871198:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3831 00000100:00080000:3.0:1787828891.871205:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828891.871211:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828891.871240:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564329600:192.168.203.139@tcp:250:kworker.0 00000008:00100000:3.0:1787828891.871243:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828891.871263:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564329728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828891.871277:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828891.871283:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828891.871285:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828891.871311:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3831 00000100:00080000:3.0:1787828891.871328:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828891.871338:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:1.0:1787828891.871339:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:3.0:1787828891.871350:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828891.871369:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564329856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828891.871383:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828891.871388:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828891.871391:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828891.871400:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3831 00000100:00100000:1.0:1787828891.871409:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828891/real 1787828891] req@ffff97688cb5e1c0 x1874670564329600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828891.871429:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828891.871443:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828891.871457:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828891.871462:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564329600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828891.871476:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828891.871483:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828891.871498:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564329600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828891.871508:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564329984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828891.871534:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564329728:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787828891.871546:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828891.871553:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828891.871558:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828891.871564:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3831 00000100:00080000:3.0:1787828891.871571:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828891.871581:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787828891.871587:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828891.871597:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828891.871621:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828891/real 1787828891] req@ffff97688cb5c780 x1874670564329728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828891.871639:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564329728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828891.871643:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564330112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828891.871699:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828891.871708:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828891.871720:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564329728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828891.871729:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564329856:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828891.871753:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828891.871761:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828891/real 1787828891] req@ffff97688cb5c3c0 x1874670564329856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828891.871769:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564329856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828891.871774:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828891.871776:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828891.871778:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564329856:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828891.871791:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564329984:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828891.871803:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828891.871810:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828891/real 1787828891] req@ffff97688cb5f0c0 x1874670564329984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828891.871817:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564329984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828891.871822:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828891.871824:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828891.871826:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564329984:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828891.871829:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564330112:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828891.871869:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828891.871877:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828891/real 1787828891] req@ffff97688cb5d2c0 x1874670564330112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828891.871884:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564330112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828891.871890:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828891.871892:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828891.871894:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564330112:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828892.126936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828892.126937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828892.127018:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828892.894940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828893.150905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828893.150936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828893.150937:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828893.918971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787828894.174893:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828894.174895:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828894.175007:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828894.943081:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828895.198903:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828895.198955:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828895.198983:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828895.966973:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828896.222952:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828896.222952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828896.223032:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828896.990903:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828896.990908:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828896.990918:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828896.990924:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828896.990935:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3836 00000100:00080000:3.0:1787828896.990947:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828896.990955:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828896.990992:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564330240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828896.991030:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828896.991033:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828896.991035:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828896.991038:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3836 00000100:00080000:3.0:1787828896.991057:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828896.991061:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828896.991069:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828896.991081:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564330368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828896.991090:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828896.991093:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564330240:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787828896.991094:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828896.991098:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828896.991105:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3836 00000100:00080000:3.0:1787828896.991107:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828896.991109:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828896.991112:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828896.991122:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564330496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828896.991129:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828896.991131:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828896.991133:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828896.991136:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3836 00000100:00080000:3.0:1787828896.991139:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828896.991141:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828896.991144:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828896.991153:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564330624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828896.991162:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787828896.991172:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828896.991176:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828896.991177:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828896.991181:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3836 00000100:00080000:3.0:1787828896.991184:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828896.991187:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828896.991190:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828896.991198:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564330752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828896.991217:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828896/real 1787828896] req@ffff97688cb5f840 x1874670564330240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828896.991238:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564330240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828896.991247:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828896.991250:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828896.991253:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564330240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828896.991274:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564330368:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828896.991293:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828896.991302:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828896/real 1787828896] req@ffff97688cb5d680 x1874670564330368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828896.991311:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564330368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828896.991317:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828896.991319:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828896.991321:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564330368:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828896.991325:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564330496:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828896.991335:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828896.991341:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828896/real 1787828896] req@ffff97688cb5de00 x1874670564330496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828896.991354:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564330496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828896.991359:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828896.991361:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828896.991363:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564330496:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828896.991366:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564330624:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828896.991377:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828896.991383:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828896/real 1787828896] req@ffff97688cb5cb40 x1874670564330624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828896.991391:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564330624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828896.991396:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828896.991398:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828896.991400:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564330624:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828896.991404:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564330752:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828896.991420:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828896.991431:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828896/real 1787828896] req@ffff97688cb5c000 x1874670564330752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828896.991444:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564330752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828896.991474:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828896.991476:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828896.991479:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564330752:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828897.246902:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828897.246921:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828897.246950:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828898.014973:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828898.270951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828898.270951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828898.271066:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828899.038923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828899.294898:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828899.294898:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828899.294943:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828900.062941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828900.318924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828900.318924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828900.319001:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828901.086917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828901.342930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828901.342945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828901.343042:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828902.110936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828902.110991:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828902.110999:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828902.111003:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828902.111016:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3842 00000100:00080000:2.0:1787828902.111027:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828902.111035:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828902.111072:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564330880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828902.111103:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828902.111107:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828902.111109:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828902.111114:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3842 00000100:00080000:2.0:1787828902.111117:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828902.111122:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828902.111132:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828902.111145:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564330880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828902.111151:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564331008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828902.111164:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828902.111168:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828902.111170:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828902.111174:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3842 00000100:00080000:2.0:1787828902.111179:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828902.111183:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828902.111188:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828902.111200:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564331136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828902.111209:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828902.111214:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828902.111218:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828902.111220:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828902.111226:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3842 00000100:00080000:2.0:1787828902.111231:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828902.111235:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828902.111241:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828902.111247:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828902/real 1787828902] req@ffff976889cfbc00 x1874670564330880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828902.111254:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564331264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828902.111266:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564330880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828902.111269:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828902.111277:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828902.111279:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828902.111286:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828902.111286:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3842 00000100:00080000:2.0:1787828902.111292:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828902.111295:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:2.0:1787828902.111299:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828902.111304:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564330880:192.168.203.139@tcp:250:kworker.0 00000008:00100000:2.0:1787828902.111306:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828902.111318:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564331392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828902.111328:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564331008:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828902.111356:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828902.111371:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828902/real 1787828902] req@ffff976889cfa580 x1874670564331008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828902.111385:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564331008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828902.111392:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828902.111394:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828902.111397:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564331008:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828902.111401:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564331136:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828902.111424:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828902.111432:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828902/real 1787828902] req@ffff976889cfb0c0 x1874670564331136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828902.111440:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564331136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828902.111445:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828902.111447:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828902.111449:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564331136:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828902.111452:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564331264:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828902.111464:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828902.111469:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828902/real 1787828902] req@ffff976889cf8000 x1874670564331264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828902.111477:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564331264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828902.111482:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828902.111483:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828902.111486:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564331264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828902.111497:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564331392:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828902.111507:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828902.111513:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828902/real 1787828902] req@ffff976889cf8f00 x1874670564331392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828902.111527:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564331392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828902.111532:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828902.111534:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828902.111536:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564331392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828902.366998:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828902.366998:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828902.367069:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828903.134960:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828903.390914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828903.390916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828903.390934:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828904.158910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828904.414972:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828904.414977:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828904.415087:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828905.182941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828905.438990:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828905.438990:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828905.439056:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828906.206918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828906.462919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828906.462920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828906.462989:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828907.230916:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828907.230927:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828907.230932:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828907.230958:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3847 00000100:00080000:2.0:1787828907.230970:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828907.230994:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828907.230999:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828907.231034:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564331520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828907.231057:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828907.231061:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828907.231063:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828907.231067:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3847 00000100:00080000:2.0:1787828907.231071:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828907.231092:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828907.231103:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828907.231118:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564331648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828907.231131:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828907.231133:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564331520:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828907.231134:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828907.231137:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828907.231141:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3847 00000100:00080000:2.0:1787828907.231145:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828907.231174:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828907.231181:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828907.231196:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564331776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828907.231209:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828907.231213:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828907.231216:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828907.231221:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3847 00000100:00080000:2.0:1787828907.231226:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828907.231231:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828907.231236:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828907.231265:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564331904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828907.231275:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828907.231279:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564331648:192.168.203.139@tcp:38:kworker.0 00000100:00080000:2.0:1787828907.231279:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828907.231281:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828907.231286:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3847 00000100:00080000:2.0:1787828907.231290:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828907.231294:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828907.231299:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828907.231327:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564331776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828907.231327:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564332032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828907.231390:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564331904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828907.231417:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564332032:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828907.486897:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828907.486903:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828907.486971:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828908.254951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828908.510921:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828908.510921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828908.510985:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828909.278968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828909.534906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828909.534953:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828909.534957:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828910.302968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:0.0:1787828910.303159:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:0.0:1787828910.303220:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828910.303246:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828910.303305:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828910.303315:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828910.303324:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787828910.303334:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:1.0:1787828910.303428:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828907/real 1787828910] req@ffff976889cf9e00 x1874670564332032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828910.303455:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564332032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828910.303469:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828910.303475:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828910.303480:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564332032:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828910.303488:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828907/real 1787828910] req@ffff976889cfad00 x1874670564331648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828910.303500:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564331648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828910.303507:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828910.303512:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828910.303516:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564331648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828910.303522:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828907/real 1787828910] req@ffff976889cfa1c0 x1874670564331776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828910.303533:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564331776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828910.303541:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828910.303545:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828910.303549:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564331776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828910.303557:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828907/real 1787828910] req@ffff976889cf92c0 x1874670564331904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828910.303567:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564331904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828910.303575:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828910.303579:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828910.303583:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564331904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828910.303589:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828907/real 1787828910] req@ffff976889cfa940 x1874670564331520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828910.303609:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564331520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828910.303617:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828910.303621:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828910.303625:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564331520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828910.558933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828910.558979:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828910.559019:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828911.327021:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828911.582919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828911.582953:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828911.582960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828912.350912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828912.350953:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828912.350959:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828912.350962:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828912.350971:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3852 00000100:00080000:3.0:1787828912.350980:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828912.350986:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828912.351021:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564332160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828912.351047:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828912.351051:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828912.351054:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828912.351058:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3852 00000100:00080000:3.0:1787828912.351063:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828912.351068:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828912.351078:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828912.351093:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564332160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828912.351095:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564332288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828912.351112:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828912.351115:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828912.351118:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828912.351123:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3852 00000100:00080000:3.0:1787828912.351128:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828912.351132:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828912.351138:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828912.351150:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564332416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828912.351160:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828912.351166:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787828912.351167:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828912.351172:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828912.351177:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3852 00000100:00080000:3.0:1787828912.351182:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828912.351187:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828912.351192:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828912.351205:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564332544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828912.351215:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828912.351219:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828912/real 1787828912] req@ffff97688cb5c000 x1874670564332160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828912.351220:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828912.351224:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828912.351233:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3852 00000100:00080000:3.0:1787828912.351241:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828912.351247:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828912.351253:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564332160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828912.351253:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828912.351270:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564332672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828912.351271:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828912.351282:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828912.351290:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564332160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828912.351320:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564332288:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828912.351348:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828912.351363:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828912/real 1787828912] req@ffff97688cb5cb40 x1874670564332288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828912.351375:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564332288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828912.351382:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828912.351385:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828912.351388:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564332288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828912.351394:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564332416:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828912.351413:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828912.351423:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828912/real 1787828912] req@ffff97688cb5de00 x1874670564332416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828912.351435:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564332416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828912.351449:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828912.351453:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828912.351456:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564332416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828912.351463:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564332544:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828912.351484:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828912.351496:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828912/real 1787828912] req@ffff97688cb5d680 x1874670564332544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828912.351507:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564332544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828912.351516:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828912.351519:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828912.351522:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564332544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828912.351528:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564332672:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828912.351548:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828912.351560:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828912/real 1787828912] req@ffff97688cb5f840 x1874670564332672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828912.351571:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564332672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828912.351580:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828912.351583:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828912.351587:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564332672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828912.606924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828912.606936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828912.607037:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828913.374931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828913.630994:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828913.630994:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828913.631002:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828914.398938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828914.654938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828914.654945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828914.655005:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828915.422950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828915.678910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828915.678913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828915.678926:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828916.446927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828916.702939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828916.702949:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828916.703013:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828917.470976:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828917.470978:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828917.470988:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828917.470993:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828917.471002:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3857 00000100:00080000:3.0:1787828917.471013:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828917.471021:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828917.471088:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564332800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828917.471117:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828917.471120:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828917.471122:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828917.471126:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3857 00000100:00080000:3.0:1787828917.471146:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828917.471149:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828917.471158:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828917.471169:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564332928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828917.471177:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828917.471180:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828917.471183:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828917.471185:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564332800:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787828917.471188:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3857 00000100:00080000:3.0:1787828917.471193:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828917.471197:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828917.471200:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828917.471210:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564333056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828917.471220:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828917.471224:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828917.471226:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828917.471241:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3857 00000100:00080000:3.0:1787828917.471246:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828917.471251:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:1.0:1787828917.471257:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:3.0:1787828917.471258:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828917.471272:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564333184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828917.471282:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828917.471284:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828917.471285:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828917.471288:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3857 00000100:00080000:3.0:1787828917.471291:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828917.471294:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828917.471298:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828917.471302:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828917/real 1787828917] req@ffff97688cb5d2c0 x1874670564332800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828917.471316:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564333312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828917.471319:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564332800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828917.471342:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828917.471345:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828917.471348:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564332800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828917.471370:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564332928:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828917.471397:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828917.471408:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828917/real 1787828917] req@ffff97688cb5f0c0 x1874670564332928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828917.471417:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564332928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828917.471423:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828917.471426:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828917.471429:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564332928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828917.471438:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564333056:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828917.471453:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828917.471460:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828917/real 1787828917] req@ffff97688cb5c3c0 x1874670564333056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828917.471488:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564333056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828917.471501:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828917.471505:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828917.471509:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564333056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828917.471515:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564333184:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828917.471535:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828917.471545:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828917/real 1787828917] req@ffff97688cb5c780 x1874670564333184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828917.471555:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564333184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828917.471562:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828917.471565:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828917.471568:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564333184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828917.471591:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564333312:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828917.471607:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828917.471615:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828917/real 1787828917] req@ffff97688cb5e1c0 x1874670564333312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828917.471623:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564333312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828917.471634:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828917.471637:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828917.471655:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564333312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828917.726943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828917.726943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828917.726989:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828918.494892:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828918.750974:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828918.750974:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828918.751054:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828919.518906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828919.774950:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828919.774971:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828919.774985:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828920.542959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828920.798991:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828920.798993:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828920.799029:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828921.566908:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828921.822933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828921.822933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828921.822960:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828922.590907:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828922.590918:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828922.590924:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828922.590928:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828922.590936:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3862 00000100:00080000:3.0:1787828922.590946:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828922.590951:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828922.590990:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564333440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828922.591019:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828922.591022:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828922.591024:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828922.591028:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3862 00000100:00080000:3.0:1787828922.591032:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828922.591035:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828922.591044:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828922.591055:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564333568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828922.591071:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828922.591074:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828922.591075:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828922.591078:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3862 00000100:00080000:3.0:1787828922.591080:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828922.591086:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828922.591087:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564333440:192.168.203.139@tcp:250:kworker.0 00000008:00100000:3.0:1787828922.591090:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828922.591102:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564333696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828922.591113:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828922.591117:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828922.591119:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828922.591124:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3862 00000100:00080000:3.0:1787828922.591129:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828922.591133:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828922.591139:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1787828922.591147:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828922.591154:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564333824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828922.591163:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828922.591166:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828922.591167:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828922.591171:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3862 00000100:00080000:3.0:1787828922.591174:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828922.591177:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828922.591181:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828922.591198:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828922/real 1787828922] req@ffff97688cb5e1c0 x1874670564333440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828922.591203:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564333952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828922.591226:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564333440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828922.591235:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828922.591239:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828922.591241:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564333440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828922.591246:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564333568:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828922.591261:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828922.591279:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828922/real 1787828922] req@ffff97688cb5c780 x1874670564333568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828922.591287:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564333568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828922.591293:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828922.591295:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828922.591297:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564333568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828922.591317:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564333696:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828922.591328:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828922.591334:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828922/real 1787828922] req@ffff97688cb5c3c0 x1874670564333696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828922.591341:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564333696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828922.591346:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828922.591348:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828922.591350:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564333696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828922.591353:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564333824:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828922.591363:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828922.591369:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828922/real 1787828922] req@ffff97688cb5f0c0 x1874670564333824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828922.591377:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564333824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828922.591381:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828922.591383:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828922.591385:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564333824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828922.591389:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564333952:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828922.591398:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828922.591412:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828922/real 1787828922] req@ffff97688cb5d2c0 x1874670564333952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828922.591418:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564333952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828922.591423:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828922.591424:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828922.591426:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564333952:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828922.846923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828922.846923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828922.846968:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828923.615014:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787828923.870970:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828923.870975:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828923.870975:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828924.638908:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828924.894933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828924.894934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828924.894996:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828925.662951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828925.918924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828925.918924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828925.918952:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828926.686980:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828926.942960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828926.942980:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828926.943078:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828927.710908:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828927.710918:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828927.710922:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828927.710931:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3867 00000100:00100000:0.0:1787828927.710942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828927.710942:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828927.710953:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828927.710992:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564334080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828927.711028:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828927.711031:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828927.711033:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828927.711038:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3867 00000100:00080000:3.0:1787828927.711042:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828927.711046:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828927.711057:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828927.711071:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564334208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828927.711083:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564334080:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787828927.711087:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828927.711092:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828927.711094:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828927.711098:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3867 00000100:00080000:3.0:1787828927.711101:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828927.711103:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828927.711107:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828927.711117:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564334336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828927.711125:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828927.711129:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828927.711130:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828927.711134:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3867 00000100:00080000:3.0:1787828927.711152:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828927.711157:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828927.711163:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1787828927.711171:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828927.711175:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564334464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828927.711192:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828927.711196:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828927.711199:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828927.711205:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3867 00000100:00080000:3.0:1787828927.711209:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828927.711217:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828927.711223:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828927.711238:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828927/real 1787828927] req@ffff97688cb5f840 x1874670564334080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828927.711242:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564334592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828927.711305:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564334080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828927.711318:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828927.711322:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828927.711326:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564334080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828927.711358:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564334208:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828927.711379:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828927.711389:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828927/real 1787828927] req@ffff97688cb5d680 x1874670564334208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828927.711397:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564334208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828927.711403:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828927.711406:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828927.711408:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564334208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828927.711412:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564334336:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787828927.711423:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828927.711430:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828927/real 1787828927] req@ffff97688cb5de00 x1874670564334336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828927.711438:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564334336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828927.711443:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828927.711445:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828927.711446:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564334336:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787828927.711450:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564334464:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828927.711461:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828927.711467:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828927/real 1787828927] req@ffff97688cb5cb40 x1874670564334464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828927.711475:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564334464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828927.711480:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828927.711482:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828927.711484:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564334464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787828927.711487:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564334592:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787828927.711497:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828927.711503:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828927/real 1787828927] req@ffff97688cb5c000 x1874670564334592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828927.711510:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564334592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828927.711516:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828927.711518:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787828927.711520:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564334592:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828927.966924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828927.966925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828927.966951:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828928.734936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828928.990919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828928.990941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828928.990973:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828929.758978:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828930.014923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828930.014923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828930.014954:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828930.782925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828931.038925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828931.038925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828931.039027:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828931.806909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828932.062939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828932.062939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828932.062978:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828932.830953:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828932.830957:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828932.830966:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828932.830970:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828932.830980:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3872 00000100:00080000:2.0:1787828932.830990:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828932.830997:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828932.831033:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564334720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828932.831060:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828932.831063:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828932.831064:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828932.831067:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3872 00000100:00080000:2.0:1787828932.831070:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828932.831073:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828932.831080:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828932.831090:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564334848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828932.831097:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828932.831100:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828932.831101:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828932.831104:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3872 00000100:00080000:2.0:1787828932.831107:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828932.831110:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564334720:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828932.831112:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828932.831120:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828932.831129:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564334976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828932.831135:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828932.831138:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828932.831139:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828932.831142:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3872 00000100:00080000:2.0:1787828932.831144:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828932.831147:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828932.831151:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828932.831157:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564335104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828932.831163:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828932.831166:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828932.831167:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828932.831171:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828932.831191:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3872 00000100:00080000:2.0:1787828932.831196:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828932.831199:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828932.831204:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828932.831227:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564335232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828932.831232:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828932/real 1787828932] req@ffff976889cfb480 x1874670564334720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828932.831271:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564334720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828932.831279:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828932.831283:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828932.831287:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564334720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828932.831293:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564334848:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828932.831311:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828932.831320:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828932/real 1787828932] req@ffff976889cfa940 x1874670564334848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828932.831329:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564334848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828932.831335:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828932.831337:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828932.831339:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564334848:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828932.831357:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564334976:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828932.831369:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828932.831375:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828932/real 1787828932] req@ffff976889cf92c0 x1874670564334976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828932.831383:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564334976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828932.831409:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828932.831411:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828932.831413:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564334976:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828932.831416:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564335104:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828932.831427:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828932.831433:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828932/real 1787828932] req@ffff976889cfa1c0 x1874670564335104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828932.831440:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564335104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828932.831445:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828932.831447:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828932.831449:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564335104:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828932.831468:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564335232:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828932.831480:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828932.831486:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828932/real 1787828932] req@ffff976889cfad00 x1874670564335232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828932.831493:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564335232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828932.831498:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828932.831500:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828932.831503:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564335232:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828933.086911:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828933.086984:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828933.087011:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828933.854885:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828934.110945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828934.110979:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828934.110991:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828934.878944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828935.134936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828935.134936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828935.135007:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828935.902959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828936.158939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828936.158939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828936.158964:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828936.926944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828937.182994:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828937.183017:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828937.183026:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828937.950897:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828937.950908:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1787828937.950910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828937.950913:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828937.950928:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3877 00000100:00080000:2.0:1787828937.950941:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828937.950949:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828937.950995:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564335360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828937.951028:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828937.951031:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828937.951033:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828937.951037:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3877 00000100:00080000:2.0:1787828937.951039:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828937.951042:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828937.951053:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828937.951065:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564335488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828937.951074:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828937.951078:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828937.951080:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828937.951083:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3877 00000100:00080000:2.0:1787828937.951086:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828937.951089:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828937.951093:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828937.951102:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564335360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828937.951104:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564335616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828937.951126:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828937.951145:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828937.951146:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828937.951150:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3877 00000100:00080000:2.0:1787828937.951153:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828937.951156:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828937.951161:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828937.951174:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564335744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828937.951183:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828937.951188:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828937.951190:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828937.951209:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3877 00000100:00080000:2.0:1787828937.951214:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828937.951220:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:1.0:1787828937.951226:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:2.0:1787828937.951231:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828937.951249:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564335872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828937.951290:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828937/real 1787828937] req@ffff976889cf9e00 x1874670564335360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828937.951314:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564335360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828937.951325:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828937.951331:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828937.951336:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564335360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828937.951346:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564335488:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828937.951378:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828937.951394:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828937/real 1787828937] req@ffff976889cf8f00 x1874670564335488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828937.951405:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564335488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828937.951413:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828937.951416:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828937.951419:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564335488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828937.951448:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564335616:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828937.951478:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828937.951508:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828937/real 1787828937] req@ffff976889cf8000 x1874670564335616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828937.951520:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564335616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828937.951527:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828937.951529:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828937.951531:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564335616:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828937.951538:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564335744:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828937.951573:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828937.951587:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828937/real 1787828937] req@ffff976889cfb0c0 x1874670564335744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828937.951604:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564335744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828937.951615:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828937.951620:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828937.951624:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564335744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828937.951632:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564335872:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828937.951651:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828937.951660:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828937/real 1787828937] req@ffff976889cfa580 x1874670564335872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828937.951670:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564335872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828937.951677:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828937.951680:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828937.951682:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564335872:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828938.206912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828938.206939:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828938.206941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828938.974892:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828939.230959:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828939.230969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828939.231062:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828939.998925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828940.254984:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828940.254991:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828940.254995:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828941.022912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828941.278960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828941.278985:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828941.279004:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828942.046894:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828942.302904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828942.302932:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828942.302939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00080000:2.0:1787828943.070939:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828943.070941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828943.070952:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828943.070958:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828943.070972:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3882 00000100:00080000:2.0:1787828943.070987:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828943.070996:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828943.071044:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564336000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828943.071074:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828943.071077:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828943.071078:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828943.071082:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3882 00000100:00080000:2.0:1787828943.071084:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828943.071087:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828943.071097:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828943.071110:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564336128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828943.071119:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828943.071123:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787828943.071125:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564336000:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828943.071126:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828943.071134:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3882 00000100:00080000:2.0:1787828943.071139:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828943.071141:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828943.071144:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828943.071152:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564336256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828943.071159:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828943.071162:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828943.071163:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828943.071167:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3882 00000100:00080000:2.0:1787828943.071169:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828943.071172:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828943.071175:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828943.071182:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564336384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828943.071188:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828943.071191:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828943.071193:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828943.071195:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3882 00000100:00080000:2.0:1787828943.071198:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828943.071200:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828943.071203:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828943.071209:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564336512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828943.071225:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828943.071296:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828943/real 1787828943] req@ffff976889cfbc00 x1874670564336000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828943.071317:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564336000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828943.071326:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828943.071330:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828943.071333:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564336000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828943.071358:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564336128:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828943.071379:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828943.071390:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828943/real 1787828943] req@ffff976889cfa580 x1874670564336128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828943.071406:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564336128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828943.071413:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828943.071415:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828943.071418:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564336128:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828943.071422:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564336256:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828943.071435:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828943.071441:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828943/real 1787828943] req@ffff976889cfb0c0 x1874670564336256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828943.071449:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564336256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828943.071455:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828943.071457:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828943.071459:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564336256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828943.071463:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564336384:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828943.071474:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828943.071479:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828943/real 1787828943] req@ffff976889cf8000 x1874670564336384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828943.071492:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564336384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828943.071499:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828943.071501:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828943.071503:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564336384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828943.071507:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564336512:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828943.071518:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828943.071524:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828943/real 1787828943] req@ffff976889cf8f00 x1874670564336512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828943.071531:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564336512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828943.071537:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828943.071539:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828943.071541:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564336512:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828943.326954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828943.326954:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828943.326981:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828944.094910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828944.350927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828944.350954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828944.350955:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828945.118929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828945.374919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828945.374923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828945.374976:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828946.142937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828946.398919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828946.398919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828946.398960:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828947.166955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828947.422911:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828947.422945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828947.422948:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828948.190926:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828948.190944:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828948.190949:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828948.190954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828948.190963:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3888 00000100:00080000:2.0:1787828948.190979:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828948.190988:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828948.191039:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564336640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828948.191087:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828948.191092:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828948.191094:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828948.191100:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3888 00000100:00080000:2.0:1787828948.191106:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828948.191112:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828948.191124:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828948.191146:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564336768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828948.191161:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828948.191166:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828948.191168:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828948.191175:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3888 00000100:00080000:2.0:1787828948.191180:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828948.191183:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564336640:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828948.191186:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828948.191199:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828948.191212:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564336896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828948.191225:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828948.191230:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828948.191232:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828948.191237:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3888 00000100:00080000:2.0:1787828948.191240:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828948.191244:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828948.191250:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828948.191263:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564337024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828948.191273:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828948.191276:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828948.191278:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828948.191282:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3888 00000100:00080000:2.0:1787828948.191287:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828948.191291:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828948.191296:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828948.191306:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564337152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828948.191318:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828948.191385:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828948/real 1787828948] req@ffff976889cf9e00 x1874670564336640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828948.191402:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564336640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828948.191414:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828948.191417:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828948.191421:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564336640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828948.191451:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564336768:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828948.191479:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828948.191495:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828948/real 1787828948] req@ffff976889cfad00 x1874670564336768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828948.191507:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564336768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828948.191514:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828948.191517:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828948.191519:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564336768:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828948.191525:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564336896:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828948.191541:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828948.191548:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828948/real 1787828948] req@ffff976889cfa1c0 x1874670564336896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828948.191556:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564336896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828948.191562:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828948.191564:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828948.191565:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564336896:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828948.191571:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564337024:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828948.191583:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828948.191589:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828948/real 1787828948] req@ffff976889cf92c0 x1874670564337024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828948.191597:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564337024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828948.191602:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828948.191604:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828948.191606:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564337024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828948.191610:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564337152:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828948.191620:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828948.191625:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828948/real 1787828948] req@ffff976889cfa940 x1874670564337152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828948.191632:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564337152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828948.191638:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828948.191640:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828948.191641:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564337152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828948.446893:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828948.446921:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828948.446925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828949.214926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828949.470969:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828949.470979:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828949.470983:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828950.238958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787828950.494933:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828950.494968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828950.494969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787828951.262903:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828951.518954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828951.518954:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828951.518980:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828952.287015:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828952.542903:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828952.542928:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828952.542934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828953.310891:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828953.310906:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828953.310918:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828953.310923:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828953.310933:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3893 00000100:00080000:2.0:1787828953.310943:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828953.310954:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828953.310992:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564337280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828953.311022:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828953.311027:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828953.311029:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828953.311034:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3893 00000100:00080000:2.0:1787828953.311039:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828953.311044:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828953.311054:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828953.311057:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564337280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828953.311071:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564337408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828953.311083:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828953.311086:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828953.311088:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828953.311091:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3893 00000100:00080000:2.0:1787828953.311096:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828953.311099:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828953.311104:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828953.311114:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828953.311118:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564337536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828953.311135:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828953.311139:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828953.311141:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828953.311146:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3893 00000100:00080000:2.0:1787828953.311150:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828953.311154:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828953.311160:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828953.311173:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564337664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828953.311178:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828953/real 1787828953] req@ffff976889cfb480 x1874670564337280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828953.311183:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828953.311190:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828953.311193:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828953.311200:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3893 00000100:00080000:2.0:1787828953.311204:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828953.311207:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828953.311212:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828953.311214:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564337280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828953.311223:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564337792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828953.311243:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828953.311248:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828953.311252:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564337280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828953.311271:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564337408:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828953.311292:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828953.311302:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828953/real 1787828953] req@ffff976889cfa940 x1874670564337408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828953.311317:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564337408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828953.311327:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828953.311338:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828953.311343:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564337408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828953.311352:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564337536:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828953.311375:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828953.311387:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828953/real 1787828953] req@ffff976889cf92c0 x1874670564337536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828953.311405:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564337536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828953.311419:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828953.311422:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828953.311426:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564337536:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828953.311433:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564337664:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828953.311449:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828953.311460:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828953/real 1787828953] req@ffff976889cfa1c0 x1874670564337664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828953.311468:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564337664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828953.311473:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828953.311475:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828953.311477:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564337664:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828953.311481:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564337792:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828953.311490:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828953.311496:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828953/real 1787828953] req@ffff976889cfad00 x1874670564337792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828953.311519:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564337792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828953.311525:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828953.311526:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828953.311528:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564337792:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828953.566950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828953.566951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828953.566977:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828954.334970:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828954.590942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828954.590972:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828954.590974:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828955.358900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828955.614943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828955.614945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828955.614994:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828956.382939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828956.638901:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828956.638901:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828956.638920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828957.406887:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828957.662882:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828957.662890:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787828957.662962:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828958.430930:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828958.430932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787828958.430941:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828958.430945:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828958.430954:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3898 00000100:00080000:3.0:1787828958.430963:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828958.430969:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828958.431006:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564337920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828958.431028:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828958.431030:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828958.431032:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828958.431041:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3898 00000100:00080000:3.0:1787828958.431044:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828958.431047:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828958.431055:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828958.431064:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564338048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828958.431072:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828958.431074:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564337920:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787828958.431075:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828958.431079:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828958.431085:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3898 00000100:00080000:3.0:1787828958.431087:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828958.431089:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828958.431092:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828958.431101:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564338176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828958.431112:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828958.431114:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828958.431116:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828958.431118:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3898 00000100:00080000:3.0:1787828958.431120:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828958.431122:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828958.431127:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1787828958.431135:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828958.431141:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564338304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828958.431152:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828958.431155:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828958.431157:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828958.431161:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3898 00000100:00080000:3.0:1787828958.431165:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828958.431186:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828958/real 1787828958] req@ffff97688cb5c000 x1874670564337920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828958.431188:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787828958.431197:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828958.431221:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564338432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828958.431222:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564337920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828958.431261:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828958.431268:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828958.431273:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564337920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828958.431304:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564338048:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828958.431333:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828958.431348:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828958/real 1787828958] req@ffff97688cb5cb40 x1874670564338048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828958.431364:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564338048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828958.431374:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828958.431378:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828958.431381:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564338048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828958.431389:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564338176:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828958.431413:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828958.431427:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828958/real 1787828958] req@ffff97688cb5de00 x1874670564338176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828958.431441:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564338176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828958.431451:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828958.431456:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828958.431459:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564338176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828958.431466:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564338304:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828958.431489:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828958.431502:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828958/real 1787828958] req@ffff97688cb5d680 x1874670564338304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828958.431516:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564338304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828958.431526:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828958.431530:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828958.431534:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564338304:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828958.431541:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564338432:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828958.431562:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828958.431574:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828958/real 1787828958] req@ffff97688cb5f840 x1874670564338432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828958.431637:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564338432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828958.431646:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828958.431650:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828958.431653:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564338432:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828958.686946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787828958.686950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828958.686967:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828959.454918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828959.710944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828959.710999:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828959.711004:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828960.478948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787828960.734917:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828960.734918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828960.734930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787828961.502898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828961.758889:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828961.758962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828961.758962:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828962.526924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828962.782926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828962.782942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828962.782951:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828963.550906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787828963.550939:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828963.550947:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828963.550952:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828963.550962:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3903 00000100:00080000:0.0:1787828963.550971:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828963.550978:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828963.551020:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564338560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828963.551046:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828963.551049:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828963.551050:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828963.551055:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3903 00000100:00080000:0.0:1787828963.551060:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828963.551065:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828963.551078:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828963.551084:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564338560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828963.551100:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564338688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828963.551116:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828963.551119:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828963.551120:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828963.551124:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3903 00000100:00080000:0.0:1787828963.551136:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828963.551140:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:1.0:1787828963.551146:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:0.0:1787828963.551149:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828963.551163:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564338816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828963.551171:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828963.551174:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828963.551175:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828963.551179:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3903 00000100:00080000:0.0:1787828963.551182:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828963.551189:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828963.551190:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828963/real 1787828963] req@ffff976882aed680 x1874670564338560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828963.551193:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787828963.551207:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564338944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828963.551216:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828963.551218:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828963.551220:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828963.551224:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3903 00000100:00100000:1.0:1787828963.551228:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564338560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828963.551229:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828963.551237:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787828963.551245:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828963.551249:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828963.551256:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564339072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828963.551256:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828963.551265:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564338560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828963.551294:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564338688:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828963.551313:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828963.551324:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828963/real 1787828963] req@ffff976882aee940 x1874670564338688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828963.551334:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564338688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828963.551349:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828963.551352:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828963.551354:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564338688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828963.551359:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564338816:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828963.551375:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828963.551388:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828963/real 1787828963] req@ffff976882aef0c0 x1874670564338816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828963.551402:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564338816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828963.551413:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828963.551417:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828963.551421:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564338816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828963.551428:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564338944:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828963.551448:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828963.551457:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828963/real 1787828963] req@ffff976882aeed00 x1874670564338944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828963.551466:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564338944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828963.551472:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828963.551474:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828963.551477:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564338944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828963.551482:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564339072:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828963.551493:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828963.551499:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828963/real 1787828963] req@ffff976882aee580 x1874670564339072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828963.551507:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564339072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828963.551513:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828963.551515:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828963.551517:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564339072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828963.806916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828963.806919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828963.806998:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828964.574907:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828964.831028:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828964.831028:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828964.831050:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828965.598913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828965.854948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828965.854962:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828965.855024:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828966.622901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828966.878944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828966.878944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828966.878977:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828967.646920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828967.902952:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828967.902964:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828967.903023:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828968.670967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828968.670967:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828968.670977:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828968.670999:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828968.671008:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3908 00000100:00080000:1.0:1787828968.671018:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828968.671024:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828968.671083:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564339200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828968.671121:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828968.671125:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828968.671127:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828968.671132:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3908 00000100:00080000:1.0:1787828968.671145:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828968.671150:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828968.671161:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828968.671168:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564339200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828968.671180:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564339328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828968.671192:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828968.671195:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828968.671197:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828968.671200:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3908 00000100:00080000:1.0:1787828968.671204:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828968.671208:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828968.671212:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828968.671224:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564339456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828968.671239:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828968.671243:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828968.671246:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828968.671253:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787828968.671255:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3908 00000100:00080000:1.0:1787828968.671261:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828968.671265:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828968.671287:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828968.671297:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564339584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828968.671308:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828968.671311:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828968.671312:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828968.671316:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3908 00000100:00080000:1.0:1787828968.671319:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828968.671321:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828968/real 1787828968] req@ffff976889efcb40 x1874670564339200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828968.671324:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828968.671330:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828968.671365:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564339712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828968.671366:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564339200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828968.671403:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828968.671407:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828968.671411:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564339200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828968.671434:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564339328:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828968.671475:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828968.671487:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828968/real 1787828968] req@ffff976889efe1c0 x1874670564339328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828968.671499:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564339328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828968.671531:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828968.671535:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828968.671538:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564339328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828968.671544:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564339456:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787828968.671580:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828968.671590:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828968/real 1787828968] req@ffff976889efe580 x1874670564339456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828968.671600:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564339456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828968.671607:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828968.671611:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828968.671616:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564339456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828968.671638:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564339584:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828968.671659:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828968.671670:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828968/real 1787828968] req@ffff976889eff480 x1874670564339584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828968.671683:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564339584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828968.671710:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828968.671714:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828968.671718:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564339584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828968.671726:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564339712:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787828968.671748:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787828968.671757:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828968/real 1787828968] req@ffff976889efc780 x1874670564339712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828968.671766:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564339712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828968.671773:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828968.671775:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828968.671778:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564339712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828968.926938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828968.926945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828968.926993:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828969.694908:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828969.950905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828969.950932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828969.950962:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828970.718952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828970.974965:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828970.974980:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828970.974982:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828971.742992:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828971.998945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828971.998967:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828971.999013:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828972.766917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828973.022916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828973.022940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828973.022946:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828973.790955:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828973.790965:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828973.790965:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828973.790970:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828973.790980:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3913 00000100:00080000:2.0:1787828973.790989:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828973.790995:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828973.791030:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564339840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828973.791053:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828973.791056:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828973.791058:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828973.791062:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3913 00000100:00080000:2.0:1787828973.791065:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828973.791068:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828973.791077:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828973.791087:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564339840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828973.791088:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564339968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828973.791109:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828973.791112:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828973.791113:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828973.791116:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3913 00000100:00080000:2.0:1787828973.791119:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828973.791122:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828973.791126:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828973.791136:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564340096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828973.791143:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828973.791146:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828973.791147:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828973.791150:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3913 00000100:00080000:2.0:1787828973.791152:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828973.791154:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828973.791158:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828973.791171:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564340224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828973.791179:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828973.791182:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828973.791183:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828973.791187:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3913 00000100:00080000:2.0:1787828973.791190:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828973.791191:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564339968:192.168.203.139@tcp:38:kworker.0 00000100:00080000:2.0:1787828973.791196:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828973.791201:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828973.791212:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564340352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828973.791223:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564340096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828973.791242:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564340224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828973.791274:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564340352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828974.046936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828974.046936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828974.047035:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828974.814955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828975.070948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828975.070980:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828975.071000:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828975.838988:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828976.094939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828976.094959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828976.094999:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828976.862923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:1.0:1787828976.863062:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:1.0:1787828976.863101:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787828976.863119:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787828976.863158:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787828976.863170:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787828976.863181:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787828976.863194:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:0.0:1787828976.863205:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828973/real 1787828976] req@ffff976889cf8f00 x1874670564339968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828976.863231:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564339968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828976.863242:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828976.863248:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828976.863252:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564339968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828976.863259:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828973/real 1787828976] req@ffff976889cf8000 x1874670564340096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828976.863270:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564340096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828976.863278:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828976.863282:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828976.863284:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564340096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787828976.863290:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828973/real 1787828976] req@ffff976889cfb0c0 x1874670564340224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828976.863302:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564340224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828976.863310:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828976.863314:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828976.863317:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564340224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828976.863324:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828973/real 1787828976] req@ffff976889cf9e00 x1874670564339840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828976.863337:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564339840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828976.863351:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828976.863356:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828976.863359:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564339840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787828976.863384:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828973/real 1787828976] req@ffff976889cfa580 x1874670564340352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828976.863397:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564340352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828976.863405:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828976.863408:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787828976.863411:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564340352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828977.118963:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828977.118963:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828977.118978:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828977.887036:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828978.142902:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828978.142904:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828978.142953:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828978.910909:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828978.910918:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828978.910922:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828978.910930:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3918 00000100:00080000:2.0:1787828978.910940:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828978.910945:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787828978.910966:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787828978.910977:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564340480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828978.911005:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828978.911008:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828978.911010:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828978.911013:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3918 00000100:00080000:2.0:1787828978.911016:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828978.911019:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828978.911027:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828978.911037:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564340608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828978.911046:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828978.911050:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787828978.911051:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564340480:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787828978.911058:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828978.911063:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3918 00000100:00080000:2.0:1787828978.911066:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828978.911068:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828978.911071:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828978.911080:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564340736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828978.911090:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828978.911097:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828978.911099:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828978.911105:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3918 00000100:00080000:2.0:1787828978.911113:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828978.911118:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828978.911123:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787828978.911129:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828978.911142:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564340864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828978.911153:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828978.911157:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828978.911159:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828978.911163:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3918 00000100:00080000:2.0:1787828978.911168:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828978.911175:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787828978.911176:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828978/real 1787828978] req@ffff976889cfbc00 x1874670564340480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828978.911180:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828978.911198:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564340992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828978.911206:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564340480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828978.911221:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828978.911227:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828978.911232:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564340480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828978.911257:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564340608:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828978.911283:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828978.911297:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828978/real 1787828978] req@ffff976889cfa580 x1874670564340608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828978.911317:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564340608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828978.911326:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828978.911329:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828978.911332:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564340608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828978.911338:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564340736:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828978.911359:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828978.911370:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828978/real 1787828978] req@ffff976889cf9e00 x1874670564340736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828978.911382:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564340736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828978.911389:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828978.911392:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828978.911395:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564340736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828978.911401:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564340864:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828978.911420:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828978.911432:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828978/real 1787828978] req@ffff976889cfb0c0 x1874670564340864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828978.911459:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564340864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828978.911474:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828978.911477:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828978.911480:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564340864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828978.911487:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564340992:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828978.911507:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828978.911517:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828978/real 1787828978] req@ffff976889cf8000 x1874670564340992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828978.911528:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564340992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828978.911537:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828978.911541:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828978.911544:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564340992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828979.166916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828979.166916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828979.166942:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828979.935074:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828980.190942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828980.190942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828980.191007:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828980.958911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828981.214913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828981.214937:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828981.214951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828981.982945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828982.238938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828982.238938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828982.239024:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828983.006895:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828983.262895:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828983.262910:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828983.262957:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828984.030931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828984.030931:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828984.030943:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828984.030948:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828984.030958:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3923 00000100:00080000:2.0:1787828984.030969:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828984.030977:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828984.031013:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564341120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828984.031043:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828984.031048:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828984.031050:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828984.031054:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3923 00000100:00080000:2.0:1787828984.031058:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828984.031063:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828984.031073:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828984.031087:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564341248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828984.031097:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828984.031099:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828984.031101:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828984.031104:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3923 00000100:00080000:2.0:1787828984.031107:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828984.031110:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828984.031114:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828984.031117:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564341120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828984.031128:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564341376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828984.031140:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828984.031143:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828984.031145:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828984.031150:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3923 00000100:00080000:2.0:1787828984.031154:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828984.031159:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828984.031165:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828984.031177:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564341504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828984.031185:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828984.031185:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828984.031191:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828984.031193:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828984.031197:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3923 00000100:00080000:2.0:1787828984.031201:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828984.031205:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828984.031208:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828984.031217:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564341632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828984.031226:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828984/real 1787828984] req@ffff976889cf8f00 x1874670564341120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828984.031244:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564341120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828984.031256:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828984.031259:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828984.031262:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564341120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828984.031267:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564341248:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828984.031281:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828984.031289:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828984/real 1787828984] req@ffff976889cfad00 x1874670564341248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828984.031297:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564341248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828984.031302:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828984.031303:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828984.031306:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564341248:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828984.031325:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564341376:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828984.031335:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828984.031341:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828984/real 1787828984] req@ffff976889cfa1c0 x1874670564341376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828984.031348:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564341376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828984.031353:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828984.031355:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828984.031357:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564341376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828984.031361:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564341504:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828984.031371:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828984.031376:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828984/real 1787828984] req@ffff976889cf92c0 x1874670564341504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828984.031384:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564341504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828984.031388:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828984.031390:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828984.031392:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564341504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828984.031395:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564341632:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828984.031426:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828984.031434:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828984/real 1787828984] req@ffff976889cfa940 x1874670564341632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828984.031442:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564341632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828984.031447:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828984.031449:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828984.031451:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564341632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828984.286935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828984.286935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828984.287009:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828985.054920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828985.310913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828985.310929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828985.310943:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828986.078948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828986.334933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828986.334933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828986.334997:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828987.102942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828987.358931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828987.358942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787828987.358950:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828988.126935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828988.382923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828988.382923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828988.382980:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828989.150897:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828989.150903:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828989.150911:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828989.150918:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828989.150928:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3929 00000100:00080000:2.0:1787828989.150936:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828989.150945:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828989.150975:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564341760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828989.150999:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828989.151002:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828989.151004:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828989.151007:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3929 00000100:00080000:2.0:1787828989.151010:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828989.151014:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828989.151023:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828989.151030:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564341760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828989.151036:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564341888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828989.151046:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828989.151049:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828989.151051:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828989.151055:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3929 00000100:00080000:2.0:1787828989.151058:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828989.151061:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828989.151065:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828989.151075:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564342016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828989.151083:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828989.151087:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787828989.151088:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828989.151092:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828989.151096:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3929 00000100:00080000:2.0:1787828989.151100:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828989.151103:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828989.151106:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787828989.151114:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564342144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828989.151121:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828989.151124:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828989/real 1787828989] req@ffff976889cfb480 x1874670564341760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828989.151125:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828989.151129:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828989.151136:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3929 00000100:00080000:2.0:1787828989.151141:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828989.151145:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828989.151149:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828989.151151:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564341760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828989.151160:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828989.151163:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828989.151166:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564341760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787828989.151184:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564341888:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828989.151201:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828989.151211:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564342272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828989.151214:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828989/real 1787828989] req@ffff976889cfa940 x1874670564341888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828989.151249:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564341888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828989.151255:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828989.151257:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828989.151260:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564341888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828989.151265:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564342016:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787828989.151279:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828989.151285:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828989/real 1787828989] req@ffff976889cf92c0 x1874670564342016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828989.151294:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564342016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828989.151299:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828989.151303:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828989.151305:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564342016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787828989.151309:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564342144:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828989.151322:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828989.151331:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828989/real 1787828989] req@ffff976889cfa1c0 x1874670564342144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828989.151340:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564342144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828989.151350:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828989.151352:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828989.151354:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564342144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828989.151390:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564342272:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787828989.151426:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787828989.151435:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828989/real 1787828989] req@ffff976889cfad00 x1874670564342272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828989.151443:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564342272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828989.151448:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828989.151450:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787828989.151452:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564342272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828989.406920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828989.406922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828989.406948:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828990.174913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828990.430988:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828990.431006:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828990.431014:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787828991.198956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787828991.454901:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828991.454910:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787828991.454973:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828992.222981:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828992.478960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828992.478995:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828992.479038:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828993.246961:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828993.502929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828993.502951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828993.502973:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828994.270901:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828994.270911:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828994.270916:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828994.270918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787828994.270927:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3934 00000100:00080000:1.0:1787828994.270942:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828994.270951:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787828994.271002:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564342400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828994.271028:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828994.271030:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828994.271032:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828994.271036:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3934 00000100:00080000:1.0:1787828994.271039:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828994.271042:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828994.271052:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828994.271062:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564342528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828994.271072:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828994.271075:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828994.271077:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828994.271081:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3934 00000100:00080000:1.0:1787828994.271085:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828994.271088:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828994.271093:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828994.271105:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564342656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828994.271116:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828994.271120:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828994.271122:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828994.271125:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3934 00000100:00080000:1.0:1787828994.271128:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828994.271131:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828994.271136:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787828994.271144:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564342784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828994.271150:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828994.271153:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828994.271154:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828994.271156:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3934 00000100:00080000:1.0:1787828994.271158:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828994.271160:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787828994.271163:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828994.271169:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564342912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828994.271277:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564342400:192.168.203.139@tcp:250:kworker.0 00000800:00000100:3.0:1787828994.271364:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828994.271405:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828994/real 1787828994] req@ffff976889efc780 x1874670564342400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828994.271419:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564342400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828994.271428:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828994.271431:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828994.271435:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564342400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787828994.271440:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564342528:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828994.271457:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828994.271465:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828994/real 1787828994] req@ffff976889eff480 x1874670564342528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828994.271473:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564342528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828994.271479:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828994.271482:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828994.271484:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564342528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828994.271487:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564342656:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828994.271498:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828994.271504:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828994/real 1787828994] req@ffff976889efe580 x1874670564342656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828994.271512:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564342656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828994.271518:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828994.271519:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828994.271521:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564342656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828994.271524:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564342784:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828994.271536:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828994.271543:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828994/real 1787828994] req@ffff976889efe1c0 x1874670564342784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828994.271551:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564342784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828994.271556:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828994.271557:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828994.271559:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564342784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828994.271562:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564342912:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828994.271573:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828994.271578:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828994/real 1787828994] req@ffff976889efcb40 x1874670564342912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828994.271586:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564342912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828994.271599:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828994.271602:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828994.271604:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564342912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787828994.526935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828994.526950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828994.527038:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828995.294950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828995.550939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828995.550960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828995.550997:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828996.318914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828996.574909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828996.574947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828996.574974:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828997.342929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787828997.598898:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828997.598898:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828997.598978:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828998.366960:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787828998.622899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787828998.622901:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787828998.622962:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828999.390916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787828999.390921:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828999.390929:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828999.390935:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828999.390947:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3939 00000100:00080000:2.0:1787828999.390957:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828999.390965:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787828999.391003:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564343040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828999.391032:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828999.391036:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828999.391038:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828999.391042:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3939 00000100:00080000:2.0:1787828999.391046:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828999.391051:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828999.391062:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828999.391073:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564343040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828999.391080:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564343168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828999.391093:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828999.391097:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828999.391100:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828999.391105:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3939 00000100:00080000:2.0:1787828999.391118:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828999.391123:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:3.0:1787828999.391128:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:2.0:1787828999.391134:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828999.391152:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564343296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828999.391164:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828999.391165:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828999/real 1787828999] req@ffff976889cf8f00 x1874670564343040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828999.391172:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828999.391177:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828999.391184:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3939 00000100:00100000:3.0:1787828999.391188:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564343040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828999.391190:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828999.391199:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:3.0:1787828999.391200:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828999.391209:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000008:00100000:2.0:1787828999.391210:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787828999.391216:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564343040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787828999.391229:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564343424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828999.391242:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828999.391246:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828999.391249:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828999.391250:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564343168:192.168.203.139@tcp:38:kworker.0 00000100:00080000:2.0:1787828999.391259:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3939 00000100:00080000:2.0:1787828999.391265:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828999.391269:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787828999.391275:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787828999.391285:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787828999.391292:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564343552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828999.391317:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828999/real 1787828999] req@ffff976889cf8000 x1874670564343168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828999.391335:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564343168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828999.391342:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828999.391345:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828999.391349:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564343168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828999.391356:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564343296:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787828999.391379:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828999.391391:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828999/real 1787828999] req@ffff976889cfb0c0 x1874670564343296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828999.391405:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564343296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828999.391413:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828999.391417:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828999.391421:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564343296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787828999.391437:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564343424:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828999.391457:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828999.391468:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828999/real 1787828999] req@ffff976889cf9e00 x1874670564343424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828999.391481:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564343424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828999.391491:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828999.391494:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828999.391498:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564343424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787828999.391505:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564343552:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787828999.391523:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787828999.391534:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828999/real 1787828999] req@ffff976889cfa580 x1874670564343552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828999.391548:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564343552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828999.391557:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828999.391561:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787828999.391565:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564343552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787828999.646936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787828999.646938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787828999.647009:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829000.414930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829000.670934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829000.670935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829000.670971:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829001.438939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829001.694968:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829001.694968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829001.695031:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829002.462949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829002.718939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829002.718973:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829002.719005:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829003.486944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829003.742930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829003.742949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829003.742985:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829004.510912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829004.510958:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829004.510966:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829004.510969:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829004.510977:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3944 00000100:00080000:3.0:1787829004.510986:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829004.510992:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829004.511023:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564343680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829004.511048:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829004.511050:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829004.511052:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829004.511055:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3944 00000100:00080000:3.0:1787829004.511057:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829004.511059:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829004.511067:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829004.511075:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564343808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829004.511087:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829004.511090:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829004.511091:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829004.511094:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3944 00000100:00080000:3.0:1787829004.511096:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829004.511098:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829004.511104:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829004.511107:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564343680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829004.511115:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564343936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829004.511123:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829004.511126:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829004.511127:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829004.511130:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3944 00000100:00080000:3.0:1787829004.511131:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829004.511133:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829004.511136:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829004.511147:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564344064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829004.511153:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829004.511156:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829004.511157:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829004.511162:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3944 00000100:00080000:3.0:1787829004.511167:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829004.511171:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829004.511177:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829004.511188:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564344192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829004.511205:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829004.511255:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829004/real 1787829004] req@ffff97688cb5d2c0 x1874670564343680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829004.511288:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564343680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829004.511299:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829004.511302:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829004.511305:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564343680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829004.511313:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564343808:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829004.511332:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829004.511342:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829004/real 1787829004] req@ffff97688cb5f0c0 x1874670564343808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829004.511350:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564343808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829004.511357:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829004.511359:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829004.511362:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564343808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829004.511391:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564343936:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829004.511402:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829004.511410:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829004/real 1787829004] req@ffff97688cb5c3c0 x1874670564343936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829004.511423:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564343936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829004.511432:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829004.511436:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829004.511440:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564343936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829004.511447:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564344064:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829004.511468:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829004.511479:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829004/real 1787829004] req@ffff97688cb5c780 x1874670564344064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829004.511488:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564344064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829004.511494:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829004.511496:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829004.511498:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564344064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829004.511502:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564344192:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829004.511515:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829004.511521:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829004/real 1787829004] req@ffff97688cb5e1c0 x1874670564344192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829004.511528:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564344192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829004.511535:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829004.511537:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829004.511539:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564344192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829004.766948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829004.766985:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829004.767062:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829005.534942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829005.790957:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829005.791008:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829005.791012:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829006.558893:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829006.814909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829006.814915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829006.815017:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829007.582923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829007.838929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829007.838949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829007.838976:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829008.606924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829008.862947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829008.862951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829008.863021:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829009.630939:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829009.630941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829009.630954:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829009.630961:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829009.630974:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3949 00000100:00080000:2.0:1787829009.630988:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829009.630998:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829009.631048:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564344320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829009.631085:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829009.631091:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829009.631094:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829009.631100:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3949 00000100:00080000:2.0:1787829009.631105:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829009.631111:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829009.631124:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829009.631136:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564344320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829009.631148:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564344448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829009.631174:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829009.631179:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829009.631181:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829009.631188:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3949 00000100:00080000:2.0:1787829009.631193:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829009.631204:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829009.631210:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829009.631219:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829009.631254:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564344576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829009.631264:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829009/real 1787829009] req@ffff976889cfbc00 x1874670564344320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829009.631274:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829009.631286:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829009.631293:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829009.631298:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564344320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829009.631302:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3949 00000100:00080000:2.0:1787829009.631314:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829009.631317:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829009.631325:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:1.0:1787829009.631326:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829009.631340:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564344320:192.168.203.139@tcp:250:kworker.0 00000008:00100000:2.0:1787829009.631343:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829009.631364:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564344704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829009.631365:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564344448:192.168.203.139@tcp:38:kworker.0 00000100:00080000:2.0:1787829009.631396:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829009.631401:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787829009.631402:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829009.631407:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829009.631418:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3949 00000100:00100000:1.0:1787829009.631419:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829009/real 1787829009] req@ffff976889cfa580 x1874670564344448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829009.631429:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829009.631435:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564344448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829009.631440:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:1.0:1787829009.631455:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787829009.631457:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829009.631461:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829009.631471:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564344448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829009.631478:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564344576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829009.631487:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564344832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829009.631496:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829009.631508:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829009/real 1787829009] req@ffff976889cf9e00 x1874670564344576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829009.631530:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564344576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829009.631541:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829009.631546:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829009.631549:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564344576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829009.631568:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564344704:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829009.631587:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829009.631595:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829009/real 1787829009] req@ffff976889cfb0c0 x1874670564344704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829009.631603:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564344704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829009.631609:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829009.631611:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829009.631614:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564344704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829009.631618:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564344832:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829009.631628:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829009.631634:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829009/real 1787829009] req@ffff976889cf8000 x1874670564344832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829009.631641:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564344832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829009.631646:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829009.631648:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829009.631650:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564344832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829009.886933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829009.886966:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829009.887017:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829010.654902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829010.910920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829010.910924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829010.910952:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829011.678932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829011.934961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829011.934973:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829011.935031:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829012.702977:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829012.958916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829012.958929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829012.958994:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829013.727049:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829013.982934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829013.982934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829013.983017:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829014.750949:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829014.750956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829014.750958:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829014.750964:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829014.750974:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3954 00000100:00080000:3.0:1787829014.750987:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829014.750996:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829014.751053:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564344960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829014.751100:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829014.751105:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829014.751107:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829014.751112:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3954 00000100:00080000:3.0:1787829014.751117:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829014.751121:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829014.751135:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829014.751152:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564345088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829014.751163:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564344960:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787829014.751182:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829014.751201:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829014.751204:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829014.751210:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3954 00000100:00080000:3.0:1787829014.751215:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829014.751220:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829014.751226:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829014.751250:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564345216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829014.751261:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829014.751265:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829014.751270:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829014.751271:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787829014.751281:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3954 00000100:00080000:3.0:1787829014.751287:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829014.751291:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829014.751297:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829014.751312:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564345344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829014.751319:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829014/real 1787829014] req@ffff97688cb5e1c0 x1874670564344960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829014.751332:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829014.751338:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829014.751340:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829014.751342:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564344960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829014.751351:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3954 00000100:00080000:1.0:1787829014.751359:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829014.751359:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829014.751367:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:3.0:1787829014.751370:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829014.751375:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564344960:192.168.203.139@tcp:250:kworker.0 00000008:00100000:3.0:1787829014.751380:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829014.751408:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564345472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829014.751419:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564345088:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829014.751439:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829014.751450:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829014/real 1787829014] req@ffff97688cb5c780 x1874670564345088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829014.751459:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564345088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829014.751465:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829014.751467:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829014.751469:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564345088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829014.751473:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564345216:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829014.751494:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829014.751500:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829014/real 1787829014] req@ffff97688cb5c3c0 x1874670564345216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829014.751507:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564345216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829014.751513:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829014.751515:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829014.751517:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564345216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829014.751520:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564345344:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829014.751530:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829014.751537:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829014/real 1787829014] req@ffff97688cb5f0c0 x1874670564345344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829014.751544:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564345344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829014.751560:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829014.751562:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829014.751564:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564345344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829014.751576:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564345472:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829014.751586:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829014.751592:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829014/real 1787829014] req@ffff97688cb5d2c0 x1874670564345472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829014.751598:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564345472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829014.751603:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829014.751605:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829014.751607:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564345472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829015.006905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829015.006933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829015.007006:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829015.774920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829016.031002:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829016.031003:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829016.031024:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829016.798962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829017.054941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829017.054941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829017.055014:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829017.822943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829018.078938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829018.078938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829018.079023:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829018.846953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829019.102994:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829019.103006:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829019.103083:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829019.870944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829019.870962:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829019.870980:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829019.870986:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829019.870998:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3959 00000100:00080000:1.0:1787829019.871013:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829019.871022:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829019.871078:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564345600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829019.871116:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829019.871119:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829019.871121:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829019.871125:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3959 00000100:00080000:1.0:1787829019.871128:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829019.871131:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829019.871141:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829019.871161:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564345600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829019.871166:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564345728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829019.871178:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829019.871187:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829019.871189:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829019.871192:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3959 00000100:00080000:1.0:1787829019.871195:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829019.871197:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829019.871201:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829019.871211:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564345856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829019.871223:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829019.871226:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829019.871227:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829019.871230:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3959 00000100:00080000:1.0:1787829019.871233:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829019.871235:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829019.871239:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829019.871255:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564345984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829019.871261:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829019.871264:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829019.871266:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787829019.871268:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787829019.871272:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3959 00000100:00080000:1.0:1787829019.871275:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829019.871278:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829019.871282:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829019.871296:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564346112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829019.871318:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829019/real 1787829019] req@ffff976889efe940 x1874670564345600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829019.871334:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564345600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829019.871343:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829019.871346:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829019.871349:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564345600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829019.871374:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564345728:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829019.871390:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829019.871411:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829019/real 1787829019] req@ffff976889eff0c0 x1874670564345728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829019.871419:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564345728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829019.871425:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829019.871427:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829019.871430:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564345728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829019.871435:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564345856:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829019.871446:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829019.871451:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829019/real 1787829019] req@ffff976889effc00 x1874670564345856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829019.871459:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564345856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829019.871476:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829019.871477:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829019.871479:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564345856:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829019.871482:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564345984:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829019.871493:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829019.871499:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829019/real 1787829019] req@ffff976889efed00 x1874670564345984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829019.871507:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564345984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829019.871512:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829019.871514:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829019.871516:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564345984:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829019.871519:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564346112:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829019.871529:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829019.871535:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829019/real 1787829019] req@ffff976889efc3c0 x1874670564346112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829019.871543:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564346112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829019.871548:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829019.871550:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829019.871551:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564346112:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829020.126913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829020.126924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829020.127003:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829020.894954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829021.150950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829021.150963:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829021.151027:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829021.918919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829022.174930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829022.174945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829022.175012:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829022.942976:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829023.198904:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829023.198930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829023.198950:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829023.966901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829024.222918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829024.222928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829024.222964:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829024.990989:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829024.991055:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829024.991064:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829024.991071:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829024.991085:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3964 00000100:00080000:3.0:1787829024.991111:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829024.991138:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829024.991198:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564346240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829024.991237:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829024.991242:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829024.991245:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829024.991251:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3964 00000100:00080000:3.0:1787829024.991256:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829024.991262:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829024.991275:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829024.991283:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564346240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829024.991302:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564346368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829024.991324:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829024.991328:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829024.991338:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829024.991384:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787829024.991429:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3964 00000100:00100000:0.0:1787829024.991432:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829024/real 1787829024] req@ffff97688cb5f840 x1874670564346240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829024.991437:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829024.991449:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829024.991457:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564346240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829024.991469:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829024.991472:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829024.991477:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564346240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829024.991508:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564346368:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829024.991530:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829024.991558:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829024/real 1787829024] req@ffff97688cb5d680 x1874670564346368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829024.991576:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564346368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829024.991585:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829024.991588:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829024.991591:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564346368:192.168.203.139@tcp:38:kworker.0 00000008:00100000:3.0:1787829024.991631:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829024.991652:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564346496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829024.991726:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829024.991732:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829024.991736:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829024.991742:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3964 00000100:00080000:3.0:1787829024.991748:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829024.991750:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564346496:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787829024.991757:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829024.991766:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829024.991784:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564346624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829024.991795:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787829024.991806:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829024.991811:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829024.991814:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829024.991820:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3964 00000100:00100000:2.0:1787829024.991821:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829024/real 1787829024] req@ffff97688cb5de00 x1874670564346496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829024.991852:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829024.991858:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829024.991874:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829024.991902:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564346496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829024.991902:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564346752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829024.991925:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829024.991932:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829024.991938:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564346496:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829024.991961:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564346624:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829024.991984:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829024.991994:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829024/real 1787829024] req@ffff97688cb5cb40 x1874670564346624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829024.992004:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564346624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829024.992011:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829024.992014:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829024.992017:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564346624:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829024.992021:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564346752:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829024.992044:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829024.992050:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829024/real 1787829024] req@ffff97688cb5c000 x1874670564346752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829024.992058:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564346752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829024.992064:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829024.992066:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829024.992068:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564346752:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829025.246871:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829025.246894:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829025.246966:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829026.014924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829026.270913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829026.270913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829026.270940:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829027.038879:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829027.294906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829027.294911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829027.294971:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829028.062920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829028.318900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829028.318914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829028.318995:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829029.086940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829029.342951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829029.342951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829029.342996:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829030.110957:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829030.110967:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1787829030.110972:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829030.110973:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829030.110986:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3970 00000100:00080000:2.0:1787829030.110999:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829030.111007:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829030.111049:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564346880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829030.111075:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829030.111089:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829030.111093:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829030.111100:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3970 00000100:00080000:2.0:1787829030.111106:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829030.111110:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829030.111121:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829030.111131:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564346880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829030.111138:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564347008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829030.111157:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829030.111160:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829030.111163:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829030.111167:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3970 00000100:00080000:2.0:1787829030.111169:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829030.111172:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829030.111175:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829030.111193:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564347136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829030.111200:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829030.111203:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829030.111204:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829030.111215:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3970 00000800:00000100:0.0:1787829030.111219:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829030.111223:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829030.111229:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829030.111235:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829030.111245:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564347264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829030.111255:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829030.111259:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829030.111261:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829030.111270:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3970 00000100:00100000:0.0:1787829030.111271:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829030/real 1787829030] req@ffff976889cf8f00 x1874670564346880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829030.111274:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829030.111291:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829030.111296:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829030.111297:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564346880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829030.111312:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564347392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829030.111313:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829030.111330:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829030.111348:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564346880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829030.111375:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564347008:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829030.111391:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829030.111401:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829030/real 1787829030] req@ffff976889cfad00 x1874670564347008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829030.111410:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564347008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829030.111415:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829030.111418:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829030.111420:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564347008:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829030.111424:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564347136:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829030.111434:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829030.111440:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829030/real 1787829030] req@ffff976889cfa1c0 x1874670564347136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829030.111447:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564347136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829030.111452:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829030.111454:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829030.111455:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564347136:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829030.111459:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564347264:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829030.111469:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829030.111475:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829030/real 1787829030] req@ffff976889cf92c0 x1874670564347264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829030.111482:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564347264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829030.111487:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829030.111489:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829030.111491:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564347264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829030.111494:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564347392:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829030.111503:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829030.111509:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829030/real 1787829030] req@ffff976889cfa940 x1874670564347392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829030.111516:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564347392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829030.111521:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829030.111522:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829030.111524:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564347392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829030.366961:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829030.366961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829030.367023:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829031.134913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829031.390974:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829031.390974:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829031.390993:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787829032.158944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829032.414906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829032.414967:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829032.414995:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829033.182966:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829033.438962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829033.439006:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829033.439013:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829034.206937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829034.462939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829034.462939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829034.463025:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829035.230898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829035.230914:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829035.230922:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829035.230927:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829035.230941:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3975 00000100:00080000:3.0:1787829035.230955:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829035.230965:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829035.231016:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564347520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829035.231057:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829035.231060:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829035.231062:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829035.231065:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3975 00000100:00080000:3.0:1787829035.231069:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829035.231073:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829035.231084:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829035.231097:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564347648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829035.231106:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829035.231108:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564347520:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787829035.231111:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829035.231116:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829035.231120:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3975 00000100:00080000:3.0:1787829035.231122:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829035.231134:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829035.231143:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829035.231157:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564347776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829035.231168:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829035.231172:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829035.231175:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829035.231179:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3975 00000100:00080000:3.0:1787829035.231183:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829035.231188:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787829035.231192:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829035.231198:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829035.231209:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564347904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829035.231216:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829035.231218:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829035.231221:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829035.231224:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3975 00000100:00080000:3.0:1787829035.231227:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829035.231231:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829035.231234:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829035.231237:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829035/real 1787829035] req@ffff97688cb5c000 x1874670564347520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829035.231246:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564348032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829035.231264:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564347520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829035.231275:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829035.231278:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829035.231281:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564347520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829035.231304:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564347648:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829035.231320:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829035.231329:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829035/real 1787829035] req@ffff97688cb5cb40 x1874670564347648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829035.231338:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564347648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829035.231344:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829035.231346:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829035.231348:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564347648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829035.231351:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564347776:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829035.231361:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829035.231367:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829035/real 1787829035] req@ffff97688cb5de00 x1874670564347776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829035.231374:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564347776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829035.231394:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829035.231396:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829035.231398:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564347776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829035.231402:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564347904:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829035.231415:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829035.231422:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829035/real 1787829035] req@ffff97688cb5d680 x1874670564347904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829035.231429:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564347904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829035.231434:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829035.231436:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829035.231438:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564347904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829035.231442:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564348032:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829035.231452:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829035.231458:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829035/real 1787829035] req@ffff97688cb5f840 x1874670564348032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829035.231465:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564348032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829035.231470:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829035.231472:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829035.231474:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564348032:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829035.486914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829035.486914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829035.486930:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829036.254918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829036.510932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829036.510978:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829036.510980:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829037.278904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829037.534901:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829037.534913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829037.534977:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829038.302953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829038.558949:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829038.558998:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829038.559013:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829039.326926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829039.582900:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829039.582907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829039.582962:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829040.350932:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829040.350936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829040.350943:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829040.350949:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829040.350959:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3980 00000100:00080000:2.0:1787829040.350971:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829040.350978:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829040.351040:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564348160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829040.351066:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829040.351069:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829040.351070:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829040.351074:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3980 00000100:00080000:2.0:1787829040.351076:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829040.351079:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829040.351089:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829040.351102:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564348288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829040.351109:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829040.351112:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829040.351113:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829040.351116:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3980 00000100:00080000:2.0:1787829040.351119:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829040.351122:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829040.351125:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829040.351131:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564348416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829040.351144:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829040.351147:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829040.351149:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829040.351157:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3980 00000100:00100000:0.0:1787829040.351158:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564348160:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829040.351161:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829040.351166:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829040.351177:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829040.351186:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564348544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829040.351192:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829040.351195:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829040.351197:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829040.351201:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3980 00000100:00080000:2.0:1787829040.351204:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829040.351206:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829040.351209:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829040.351217:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564348672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829040.351301:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564348288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829040.351354:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564348416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829040.351390:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564348544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829040.351433:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564348672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829040.606928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829040.606964:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829040.606982:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829041.374922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829041.630938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829041.630944:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829041.630953:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829042.398947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829042.654941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829042.654941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829042.655005:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829043.422901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:1.0:1787829043.423102:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:1.0:1787829043.423154:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787829043.423176:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787829043.423227:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787829043.423241:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787829043.423255:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787829043.423267:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:0.0:1787829043.423277:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829040/real 1787829043] req@ffff976889cf92c0 x1874670564348416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829043.423309:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564348416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829043.423324:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829043.423330:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829043.423335:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564348416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829043.423344:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829040/real 1787829043] req@ffff976889cfa1c0 x1874670564348544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829043.423364:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564348544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829043.423373:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829043.423376:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829043.423379:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564348544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829043.423387:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829040/real 1787829043] req@ffff976889cfad00 x1874670564348672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829043.423398:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564348672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829043.423406:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829043.423409:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829043.423413:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564348672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829043.423420:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829040/real 1787829043] req@ffff976889cfb480 x1874670564348160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829043.423432:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564348160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829043.423440:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829043.423443:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829043.423447:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564348160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829043.423455:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829040/real 1787829043] req@ffff976889cfa940 x1874670564348288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829043.423469:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564348288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829043.423478:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829043.423482:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829043.423486:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564348288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829043.678962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829043.678963:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829043.679092:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829044.446943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829044.702975:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829044.703027:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829044.703123:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829045.470904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829045.470959:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829045.470965:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829045.470970:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829045.470994:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3985 00000100:00080000:0.0:1787829045.471006:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829045.471013:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829045.471057:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564348800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829045.471097:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829045.471102:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829045.471104:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829045.471109:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3985 00000100:00080000:0.0:1787829045.471114:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829045.471119:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829045.471132:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829045.471151:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564348928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829045.471157:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564348800:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829045.471162:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829045.471168:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829045.471172:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829045.471178:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3985 00000100:00080000:0.0:1787829045.471182:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829045.471187:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829045.471193:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829045.471206:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564349056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829045.471216:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829045.471220:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829045.471228:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829045.471233:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3985 00000100:00080000:0.0:1787829045.471238:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829045.471242:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829045.471248:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829045.471263:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564349184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829045.471273:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829045.471276:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829045.471280:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829045.471283:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829045.471290:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3985 00000100:00080000:0.0:1787829045.471294:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829045.471298:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829045.471304:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829045.471318:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564349312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829045.471341:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829045/real 1787829045] req@ffff976882aec3c0 x1874670564348800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829045.471366:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564348800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829045.471377:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829045.471382:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829045.471387:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564348800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829045.471417:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564348928:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829045.471443:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829045.471459:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829045/real 1787829045] req@ffff976882aee580 x1874670564348928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829045.471472:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564348928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829045.471481:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829045.471485:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829045.471488:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564348928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829045.471494:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564349056:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829045.471517:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829045.471530:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829045/real 1787829045] req@ffff976882aeed00 x1874670564349056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829045.471550:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564349056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829045.471558:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829045.471562:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829045.471566:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564349056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829045.471572:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564349184:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829045.471596:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829045.471608:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829045/real 1787829045] req@ffff976882aef0c0 x1874670564349184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829045.471620:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564349184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829045.471630:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829045.471633:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829045.471636:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564349184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829045.471642:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564349312:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829045.471662:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829045.471672:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829045/real 1787829045] req@ffff976882aee940 x1874670564349312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829045.471683:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564349312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829045.471696:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829045.471700:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829045.471703:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564349312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829045.726935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829045.727002:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829045.727018:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829046.494906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829046.750945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829046.750945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829046.750948:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829047.518957:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829047.774924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829047.774924:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829047.774926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829048.542931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829048.798930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829048.798986:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829048.798996:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829049.566940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829049.822917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829049.822917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829049.822997:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829050.590911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829050.590912:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829050.590922:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829050.590926:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829050.590936:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3990 00000100:00080000:0.0:1787829050.590961:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829050.590970:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829050.591013:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564349440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829050.591039:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829050.591042:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829050.591044:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829050.591047:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3990 00000100:00080000:0.0:1787829050.591049:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829050.591052:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829050.591062:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829050.591075:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564349568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829050.591085:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564349440:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829050.591086:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829050.591093:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829050.591096:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829050.591100:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3990 00000100:00080000:0.0:1787829050.591102:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829050.591104:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829050.591107:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829050.591116:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564349696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829050.591123:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829050.591125:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829050.591126:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829050.591130:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3990 00000100:00080000:0.0:1787829050.591132:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829050.591136:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829050.591139:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829050.591147:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564349824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829050.591154:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829050.591157:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829050.591158:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829050.591161:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3990 00000100:00080000:0.0:1787829050.591164:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829050.591169:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:2.0:1787829050.591170:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:0.0:1787829050.591175:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829050.591184:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564349952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829050.591224:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829050/real 1787829050] req@ffff976882aed680 x1874670564349440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829050.591239:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564349440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829050.591253:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829050.591260:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829050.591264:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564349440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829050.591287:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564349568:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829050.591304:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829050.591314:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829050/real 1787829050] req@ffff976882aee940 x1874670564349568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829050.591323:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564349568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829050.591341:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829050.591344:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829050.591346:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564349568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829050.591351:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564349696:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829050.591363:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829050.591370:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829050/real 1787829050] req@ffff976882aef0c0 x1874670564349696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829050.591378:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564349696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829050.591383:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829050.591386:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829050.591388:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564349696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829050.591391:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564349824:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829050.591408:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829050.591414:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829050/real 1787829050] req@ffff976882aeed00 x1874670564349824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829050.591422:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564349824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829050.591427:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829050.591429:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829050.591432:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564349824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829050.591435:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564349952:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829050.591446:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829050.591451:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829050/real 1787829050] req@ffff976882aee580 x1874670564349952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829050.591458:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564349952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829050.591464:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829050.591466:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829050.591467:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564349952:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829050.846953:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829050.846989:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829050.847041:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829051.614891:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829051.870909:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829051.870958:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829051.870971:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829052.638887:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829052.894900:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829052.894902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829052.895009:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829053.662977:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829053.918959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829053.918983:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829053.918997:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829054.686929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829054.942969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829054.942983:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829054.943113:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829055.710969:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829055.710986:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829055.710995:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829055.711001:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829055.711011:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 3995 00000100:00080000:2.0:1787829055.711024:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829055.711031:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829055.711073:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564350080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829055.711104:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829055.711108:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829055.711109:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829055.711113:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3995 00000100:00080000:2.0:1787829055.711116:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829055.711122:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829055.711133:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829055.711150:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564350080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829055.711150:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564350208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829055.711181:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829055.711185:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829055.711194:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829055.711201:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3995 00000100:00080000:2.0:1787829055.711206:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829055.711213:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829055.711220:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829055.711233:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564350336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829055.711242:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829055.711245:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829055.711246:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829055.711256:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829055.711263:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3995 00000100:00080000:2.0:1787829055.711268:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829055.711273:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829055.711279:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829055.711293:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564350464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829055.711305:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829055.711307:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829055/real 1787829055] req@ffff976889cf8f00 x1874670564350080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829055.711314:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829055.711325:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829055.711338:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 3995 00000100:00100000:0.0:1787829055.711340:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564350080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829055.711350:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829055.711361:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829055.711363:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:0.0:1787829055.711372:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000008:00100000:2.0:1787829055.711375:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829055.711378:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564350080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829055.711395:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564350592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829055.711409:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564350208:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829055.711430:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829055.711445:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829055/real 1787829055] req@ffff976889cf8000 x1874670564350208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829055.711458:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564350208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829055.711464:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829055.711466:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829055.711468:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564350208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829055.711473:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564350336:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829055.711483:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829055.711489:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829055/real 1787829055] req@ffff976889cfb0c0 x1874670564350336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829055.711496:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564350336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829055.711502:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829055.711503:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829055.711505:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564350336:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829055.711509:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564350464:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829055.711520:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829055.711526:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829055/real 1787829055] req@ffff976889cf9e00 x1874670564350464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829055.711541:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564350464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829055.711550:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829055.711553:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829055.711555:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564350464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829055.711565:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564350592:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829055.711577:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829055.711583:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829055/real 1787829055] req@ffff976889cfa580 x1874670564350592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829055.711591:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564350592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829055.711596:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829055.711598:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829055.711600:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564350592:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829055.966981:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829055.966982:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829055.967042:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829056.734973:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829056.990936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829056.990936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829056.991011:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829057.758995:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829058.014960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829058.014967:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829058.015024:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829058.782945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829059.038908:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829059.038910:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829059.038988:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829059.806942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829060.062947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829060.062947:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829060.063016:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829060.830915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829060.830935:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829060.830943:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829060.830948:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829060.830958:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4000 00000100:00080000:2.0:1787829060.830977:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829060.830985:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829060.831022:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564350720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829060.831058:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829060.831062:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829060.831064:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829060.831068:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4000 00000100:00080000:2.0:1787829060.831071:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829060.831076:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829060.831086:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829060.831101:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564350848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829060.831106:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564350720:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829060.831121:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829060.831125:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829060.831127:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829060.831132:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4000 00000100:00080000:2.0:1787829060.831136:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829060.831140:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829060.831145:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829060.831156:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564350976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829060.831167:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829060.831168:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829060.831173:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829060.831176:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829060.831182:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4000 00000100:00080000:2.0:1787829060.831186:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829060.831191:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829060.831196:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829060.831208:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564351104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829060.831210:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829060/real 1787829060] req@ffff976889cfbc00 x1874670564350720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829060.831223:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829060.831231:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829060.831234:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829060.831243:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4000 00000100:00080000:2.0:1787829060.831249:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829060.831255:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829060.831260:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829060.831268:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564350720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829060.831273:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564351232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829060.831294:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829060.831300:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829060.831304:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564350720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829060.831325:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564350848:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829060.831346:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829060.831357:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829060/real 1787829060] req@ffff976889cfa580 x1874670564350848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829060.831368:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564350848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829060.831382:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829060.831386:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829060.831388:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564350848:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829060.831393:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564350976:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829060.831413:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829060.831422:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829060/real 1787829060] req@ffff976889cf9e00 x1874670564350976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829060.831430:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564350976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829060.831447:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829060.831450:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829060.831452:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564350976:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829060.831456:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564351104:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829060.831472:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829060.831484:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829060/real 1787829060] req@ffff976889cfb0c0 x1874670564351104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829060.831494:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564351104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829060.831503:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829060.831507:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829060.831512:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564351104:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829060.831520:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564351232:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829060.831545:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829060.831559:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829060/real 1787829060] req@ffff976889cf8000 x1874670564351232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829060.831569:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564351232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829060.831575:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829060.831578:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829060.831582:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564351232:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829061.086902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829061.086910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829061.086958:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829061.854950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829062.110935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829062.110960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829062.111008:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829062.878904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829063.134930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829063.134930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829063.134981:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829063.902929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829064.158991:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829064.158992:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829064.159151:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829064.926913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829065.182958:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829065.182998:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829065.183069:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829065.950916:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829065.950916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829065.950927:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829065.950931:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829065.950939:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4005 00000100:00080000:2.0:1787829065.950947:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829065.950953:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829065.950985:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564351360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829065.951008:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829065.951011:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829065.951012:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829065.951015:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4005 00000100:00080000:2.0:1787829065.951017:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829065.951020:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829065.951027:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829065.951037:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564351488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829065.951056:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829065.951062:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787829065.951064:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564351360:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829065.951065:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829065.951075:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4005 00000100:00080000:2.0:1787829065.951079:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829065.951084:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829065.951090:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829065.951104:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564351616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829065.951113:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829065.951115:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829065.951117:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829065.951121:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4005 00000800:00000100:0.0:1787829065.951126:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829065.951128:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829065.951134:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829065.951138:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829065.951148:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564351744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829065.951154:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829065.951157:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829065.951158:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829065.951161:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4005 00000100:00080000:2.0:1787829065.951163:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829065.951167:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829065/real 1787829065] req@ffff976889cf8f00 x1874670564351360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829065.951169:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829065.951175:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829065.951187:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564351872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829065.951192:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564351360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829065.951206:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829065.951209:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829065.951212:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564351360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829065.951231:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564351488:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829065.951246:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829065.951255:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829065/real 1787829065] req@ffff976889cfa940 x1874670564351488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829065.951263:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564351488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829065.951268:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829065.951271:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829065.951274:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564351488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829065.951277:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564351616:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829065.951287:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829065.951293:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829065/real 1787829065] req@ffff976889cfb480 x1874670564351616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829065.951300:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564351616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829065.951305:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829065.951307:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829065.951309:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564351616:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829065.951312:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564351744:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829065.951322:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829065.951327:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829065/real 1787829065] req@ffff976889cfad00 x1874670564351744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829065.951335:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564351744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829065.951340:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829065.951341:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829065.951343:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564351744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829065.951346:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564351872:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829065.951355:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829065.951361:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829065/real 1787829065] req@ffff976889cfa1c0 x1874670564351872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829065.951368:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564351872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829065.951373:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829065.951375:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829065.951384:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564351872:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829066.206944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829066.206945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829066.207056:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829066.974931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829067.230954:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829067.230974:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829067.231002:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829067.998944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829068.254950:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829068.254959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829068.255036:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829069.022962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829069.278934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829069.278960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829069.279035:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829070.046943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829070.302954:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829070.302967:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829070.303043:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829071.070985:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829071.070996:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829071.071000:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829071.071012:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829071.071028:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4010 00000100:00080000:0.0:1787829071.071046:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829071.071059:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829071.071139:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564352000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829071.071188:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829071.071193:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829071.071196:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829071.071203:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4010 00000100:00080000:0.0:1787829071.071209:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829071.071218:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829071.071234:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829071.071261:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564352128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829071.071278:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829071.071282:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564352000:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829071.071283:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829071.071291:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829071.071308:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4010 00000100:00080000:0.0:1787829071.071313:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829071.071319:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829071.071327:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829071.071365:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564352256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829071.071372:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829071.071396:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829071.071402:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829071.071406:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829071.071417:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4010 00000100:00080000:0.0:1787829071.071424:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829071.071432:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829071/real 1787829071] req@ffff976882aec3c0 x1874670564352000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829071.071434:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829071.071458:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829071.071474:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564352000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829071.071485:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829071.071492:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564352384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829071.071493:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829071.071500:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564352000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829071.071534:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564352128:192.168.203.139@tcp:38:kworker.0 00000100:00080000:0.0:1787829071.071535:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829071.071547:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829071.071552:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829071.071559:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4010 00000800:00000100:2.0:1787829071.071564:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829071.071570:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829071.071593:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829071/real 1787829071] req@ffff976882aee580 x1874670564352128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829071.071600:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829071.071621:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829071.071628:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564352128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829071.071637:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829071.071642:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829071.071649:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564352512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829071.071657:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564352128:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829071.071687:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564352256:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829071.071710:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829071.071724:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829071/real 1787829071] req@ffff976882aeed00 x1874670564352256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829071.071732:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564352256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829071.071741:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829071.071744:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829071.071746:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564352256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829071.071758:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564352384:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829071.071769:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829071.071775:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829071/real 1787829071] req@ffff976882aef0c0 x1874670564352384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829071.071782:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564352384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829071.071787:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829071.071789:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829071.071791:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564352384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829071.071816:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564352512:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829071.071863:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829071.071872:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829071/real 1787829071] req@ffff976882aee940 x1874670564352512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829071.071879:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564352512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829071.071894:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829071.071896:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829071.071900:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564352512:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829071.326962:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829071.327024:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829071.327035:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829072.094928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829072.350973:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829072.351000:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829072.351016:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829073.118974:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829073.374978:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829073.375002:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829073.375025:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829074.142897:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829074.398968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829074.399070:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829074.399083:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829075.166959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829075.422954:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829075.422958:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829075.423022:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829076.190936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829076.190941:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829076.190950:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829076.190955:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829076.190964:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4016 00000100:00080000:1.0:1787829076.190977:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829076.190985:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829076.191038:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564352640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829076.191062:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829076.191065:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829076.191067:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829076.191073:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4016 00000100:00080000:1.0:1787829076.191077:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829076.191080:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829076.191092:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829076.191105:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564352768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829076.191113:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829076.191116:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829076.191117:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829076.191120:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4016 00000100:00080000:1.0:1787829076.191123:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829076.191125:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829076.191129:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829076.191137:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564352896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829076.191145:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829076.191149:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829076.191153:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829076.191156:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564352640:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829076.191157:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4016 00000100:00080000:1.0:1787829076.191162:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829076.191171:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829076.191176:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829076.191188:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564353024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829076.191198:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829076.191201:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829076.191202:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829076.191210:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4016 00000100:00080000:1.0:1787829076.191213:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829076.191216:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829076.191219:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829076.191230:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564353152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829076.191282:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829076.191335:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829076/real 1787829076] req@ffff976889efc3c0 x1874670564352640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829076.191355:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564352640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829076.191369:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829076.191373:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829076.191376:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564352640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829076.191382:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564352768:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829076.191403:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829076.191413:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829076/real 1787829076] req@ffff976889efed00 x1874670564352768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829076.191421:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564352768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829076.191427:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829076.191430:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829076.191432:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564352768:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829076.191454:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564352896:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829076.191466:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829076.191480:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829076/real 1787829076] req@ffff976889effc00 x1874670564352896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829076.191488:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564352896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829076.191494:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829076.191496:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829076.191498:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564352896:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829076.191502:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564353024:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829076.191515:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829076.191521:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829076/real 1787829076] req@ffff976889eff0c0 x1874670564353024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829076.191528:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564353024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829076.191534:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829076.191535:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829076.191538:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564353024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829076.191545:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564353152:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829076.191575:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829076.191587:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829076/real 1787829076] req@ffff976889efe940 x1874670564353152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829076.191599:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564353152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829076.191608:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829076.191611:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829076.191615:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564353152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829076.446939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829076.446940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829076.446999:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829077.214950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829077.470947:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829077.470983:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829077.471012:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787829078.238910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829078.494922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829078.494940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829078.495012:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829079.262974:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829079.518964:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829079.518964:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829079.519066:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829080.286946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829080.542906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829080.542972:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829080.543014:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829081.310914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829081.310960:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829081.310966:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829081.310972:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829081.310985:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4021 00000100:00080000:0.0:1787829081.311008:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829081.311019:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829081.311072:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564353280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829081.311108:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829081.311111:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829081.311113:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829081.311117:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4021 00000100:00080000:0.0:1787829081.311122:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829081.311128:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829081.311142:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829081.311162:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564353408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829081.311178:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829081.311182:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564353280:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829081.311184:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829081.311190:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829081.311199:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4021 00000100:00080000:0.0:1787829081.311203:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829081.311209:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829081.311215:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829081.311227:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564353536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829081.311236:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829081.311239:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829081.311241:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829081.311245:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4021 00000100:00080000:0.0:1787829081.311249:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829081.311255:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:2.0:1787829081.311258:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:0.0:1787829081.311265:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829081.311280:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564353664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829081.311289:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829081.311304:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829081.311307:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829081.311310:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4021 00000100:00080000:0.0:1787829081.311313:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829081.311316:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829081.311321:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829081.311332:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564353792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829081.311339:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829081/real 1787829081] req@ffff976882aed680 x1874670564353280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829081.311369:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564353280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829081.311380:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829081.311384:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829081.311388:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564353280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829081.311427:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564353408:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829081.311447:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829081.311457:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829081/real 1787829081] req@ffff976882aee940 x1874670564353408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829081.311465:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564353408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829081.311471:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829081.311474:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829081.311476:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564353408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829081.311480:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564353536:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829081.311491:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829081.311497:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829081/real 1787829081] req@ffff976882aef0c0 x1874670564353536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829081.311504:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564353536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829081.311509:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829081.311511:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829081.311513:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564353536:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829081.311517:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564353664:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829081.311528:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829081.311534:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829081/real 1787829081] req@ffff976882aeed00 x1874670564353664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829081.311546:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564353664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829081.311551:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829081.311553:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829081.311555:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564353664:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829081.311559:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564353792:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829081.311596:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829081.311605:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829081/real 1787829081] req@ffff976882aee580 x1874670564353792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829081.311615:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564353792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829081.311622:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829081.311625:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829081.311628:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564353792:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829081.566932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829081.566964:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829081.567116:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829082.334953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829082.590922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829082.590933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829082.590979:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829083.358970:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829083.614880:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829083.614920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829083.614985:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829084.382946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829084.638981:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829084.638985:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829084.639079:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829085.406939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829085.662925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829085.662925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829085.662995:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829086.430946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829086.430957:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829086.430966:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829086.430972:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829086.430985:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4026 00000100:00080000:1.0:1787829086.431002:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829086.431011:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829086.431059:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564353920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829086.431091:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829086.431095:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829086.431097:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829086.431110:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4026 00000100:00080000:1.0:1787829086.431119:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829086.431125:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829086.431138:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829086.431146:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564353920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829086.431156:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564354048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829086.431168:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829086.431172:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829086.431174:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829086.431179:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4026 00000100:00080000:1.0:1787829086.431183:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829086.431187:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829086.431192:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829086.431203:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564354176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829086.431213:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829086.431217:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829086.431218:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829086.431223:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4026 00000100:00080000:1.0:1787829086.431226:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829086.431229:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829086.431234:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1787829086.431242:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829086.431252:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564354304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829086.431263:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829086.431266:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829086.431267:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829086.431270:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4026 00000100:00080000:1.0:1787829086.431275:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829086.431280:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829086.431285:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829086.431289:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829086/real 1787829086] req@ffff976889efcb40 x1874670564353920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829086.431303:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564354432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829086.431307:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564353920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829086.431333:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829086.431338:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829086.431341:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564353920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829086.431366:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564354048:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829086.431383:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829086.431393:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829086/real 1787829086] req@ffff976889efe1c0 x1874670564354048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829086.431402:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564354048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829086.431408:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829086.431410:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829086.431413:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564354048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829086.431416:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564354176:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829086.431427:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829086.431442:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829086/real 1787829086] req@ffff976889efe580 x1874670564354176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829086.431454:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564354176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829086.431464:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829086.431466:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829086.431469:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564354176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829086.431472:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564354304:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829086.431485:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829086.431491:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829086/real 1787829086] req@ffff976889eff480 x1874670564354304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829086.431498:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564354304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829086.431504:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829086.431506:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829086.431508:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564354304:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829086.431512:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564354432:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829086.431522:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829086.431527:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829086/real 1787829086] req@ffff976889efc780 x1874670564354432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829086.431534:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564354432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829086.431540:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829086.431542:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829086.431544:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564354432:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829086.686918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829086.686918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829086.686982:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829087.454904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829087.710910:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829087.710948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829087.711064:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829088.478987:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829088.734937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829088.734952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829088.734995:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829089.502902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829089.758884:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829089.758938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829089.758949:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829090.526958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829090.782885:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829090.782885:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829090.782984:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829091.550911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829091.550915:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829091.550922:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829091.550927:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829091.550937:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4031 00000100:00080000:1.0:1787829091.550950:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829091.550958:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829091.551018:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564354560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829091.551040:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829091.551043:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829091.551044:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829091.551048:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4031 00000100:00080000:1.0:1787829091.551051:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829091.551054:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829091.551063:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829091.551073:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564354688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829091.551081:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564354560:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829091.551083:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829091.551089:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829091.551090:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829091.551093:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4031 00000100:00080000:1.0:1787829091.551095:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829091.551097:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829091.551100:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829091.551109:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564354816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829091.551115:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829091.551118:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829091.551120:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829091.551124:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4031 00000100:00080000:1.0:1787829091.551127:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829091.551130:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829091.551133:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829091.551141:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564354944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829091.551147:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829091.551149:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829091.551151:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829091.551153:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4031 00000100:00080000:1.0:1787829091.551155:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829091.551157:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829091.551160:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829091.551163:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829091.551170:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564355072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829091.551224:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829091/real 1787829091] req@ffff976889efc780 x1874670564354560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829091.551247:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564354560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829091.551261:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829091.551265:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829091.551270:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564354560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829091.551297:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564354688:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829091.551324:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829091.551335:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829091/real 1787829091] req@ffff976889eff480 x1874670564354688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829091.551344:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564354688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829091.551351:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829091.551353:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829091.551357:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564354688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829091.551362:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564354816:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829091.551382:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829091.551394:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829091/real 1787829091] req@ffff976889efe580 x1874670564354816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829091.551407:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564354816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829091.551416:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829091.551420:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829091.551424:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564354816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829091.551431:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564354944:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829091.551451:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829091.551462:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829091/real 1787829091] req@ffff976889efe1c0 x1874670564354944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829091.551471:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564354944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829091.551477:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829091.551479:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829091.551481:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564354944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829091.551486:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564355072:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829091.551499:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829091.551506:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829091/real 1787829091] req@ffff976889efcb40 x1874670564355072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829091.551514:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564355072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829091.551519:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829091.551521:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829091.551523:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564355072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829091.806893:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829091.806961:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829091.806983:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829092.574937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829092.830888:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829092.830918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829092.830952:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829093.598882:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829093.854892:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829093.854916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829093.854984:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829094.622916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829094.878901:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829094.878952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829094.878982:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829095.646951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829095.902903:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829095.902912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829095.903004:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829096.670972:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829096.670975:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829096.670991:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829096.670995:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829096.671005:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4036 00000100:00080000:0.0:1787829096.671014:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829096.671020:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829096.671053:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564355200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829096.671082:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829096.671085:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829096.671087:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829096.671090:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4036 00000100:00080000:0.0:1787829096.671092:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829096.671095:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829096.671103:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829096.671116:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564355328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829096.671128:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829096.671132:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829096.671134:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829096.671141:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564355200:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829096.671145:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4036 00000100:00080000:0.0:1787829096.671158:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829096.671167:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829096.671174:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829096.671193:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564355456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829096.671202:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829096.671205:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829096.671206:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829096.671212:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4036 00000100:00080000:0.0:1787829096.671220:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829096.671224:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829096.671229:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829096.671237:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564355584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829096.671245:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829096.671247:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829096.671248:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829096.671250:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4036 00000100:00080000:0.0:1787829096.671254:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829096.671264:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829096.671265:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829096.671275:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829096.671284:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564355712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829096.671658:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829096/real 1787829096] req@ffff976882aec3c0 x1874670564355200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829096.671685:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564355200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829096.671700:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829096.671706:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829096.671711:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564355200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829096.671744:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564355328:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829096.671775:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829096.671791:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829096/real 1787829096] req@ffff976882aee580 x1874670564355328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829096.671807:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564355328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829096.671816:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829096.671821:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829096.671859:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564355328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829096.671869:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564355456:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829096.671904:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829096.671918:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829096/real 1787829096] req@ffff976882aeed00 x1874670564355456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829096.671932:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564355456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829096.671942:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829096.671946:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829096.671950:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564355456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829096.671957:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564355584:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829096.671981:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829096.671995:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829096/real 1787829096] req@ffff976882aef0c0 x1874670564355584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829096.672009:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564355584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829096.672020:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829096.672024:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829096.672028:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564355584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829096.672035:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564355712:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829096.672058:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829096.672070:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829096/real 1787829096] req@ffff976882aee940 x1874670564355712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829096.672084:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564355712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829096.672094:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829096.672098:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829096.672102:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564355712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829096.926957:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829096.926957:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829096.927045:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829097.694937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829097.950936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829097.950936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829097.951036:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829098.718963:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829098.974950:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829098.974953:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829098.975000:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829099.742908:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829099.998897:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829099.998901:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829099.998984:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829100.766953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829101.022913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829101.022913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829101.023005:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829101.790967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829101.790971:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829101.790982:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829101.790987:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829101.791001:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4041 00000100:00080000:1.0:1787829101.791013:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829101.791022:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829101.791065:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564355840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829101.791088:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829101.791092:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829101.791093:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829101.791097:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4041 00000100:00080000:1.0:1787829101.791099:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829101.791101:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829101.791112:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829101.791123:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564355968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829101.791131:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829101.791133:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829101.791135:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829101.791140:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4041 00000100:00100000:3.0:1787829101.791141:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564355840:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829101.791142:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829101.791147:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829101.791159:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829101.791166:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564356096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829101.791174:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829101.791176:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829101.791178:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829101.791181:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4041 00000100:00080000:1.0:1787829101.791183:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829101.791185:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829101.791188:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829101.791198:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564356224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829101.791204:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829101.791208:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829101.791209:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829101.791212:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4041 00000100:00080000:1.0:1787829101.791216:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829101.791220:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829101.791224:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829101.791235:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564356352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829101.791249:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829101.791305:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829101/real 1787829101] req@ffff976889efe940 x1874670564355840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829101.791329:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564355840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829101.791341:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829101.791346:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829101.791351:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564355840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829101.791381:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564355968:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829101.791408:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829101.791423:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829101/real 1787829101] req@ffff976889eff0c0 x1874670564355968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829101.791437:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564355968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829101.791447:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829101.791451:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829101.791455:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564355968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829101.791461:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564356096:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829101.791483:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829101.791496:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829101/real 1787829101] req@ffff976889effc00 x1874670564356096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829101.791509:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564356096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829101.791524:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829101.791528:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829101.791531:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564356096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829101.791538:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564356224:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829101.791561:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829101.791573:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829101/real 1787829101] req@ffff976889efed00 x1874670564356224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829101.791602:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564356224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829101.791611:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829101.791615:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829101.791618:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564356224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829101.791634:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564356352:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829101.791657:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829101.791670:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829101/real 1787829101] req@ffff976889efc3c0 x1874670564356352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829101.791682:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564356352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829101.791691:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829101.791694:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829101.791698:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564356352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829102.046920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829102.046920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829102.046999:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829102.814904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829103.070960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829103.070960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829103.071011:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829103.838912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829104.094919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829104.094919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829104.094984:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829104.862920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829105.118975:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829105.118976:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829105.119051:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829105.886891:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829106.142921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829106.142921:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829106.143007:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829106.910908:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829106.910950:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829106.910959:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829106.910964:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829106.910977:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4046 00000100:00080000:1.0:1787829106.910992:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829106.911001:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829106.911063:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564356480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829106.911093:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829106.911097:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829106.911100:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829106.911106:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4046 00000100:00080000:1.0:1787829106.911111:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829106.911117:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829106.911126:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564356480:192.168.203.139@tcp:250:kworker.0 00000008:00100000:1.0:1787829106.911129:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829106.911149:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564356608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829106.911162:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829106.911165:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829106.911167:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829106.911172:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4046 00000100:00080000:1.0:1787829106.911177:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829106.911181:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829106.911187:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829106.911202:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564356736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829106.911213:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829106.911216:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829106.911218:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829106.911223:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4046 00000100:00080000:1.0:1787829106.911227:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829106.911232:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829106.911239:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829106.911250:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564356608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829106.911256:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564356864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829106.911270:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829106.911274:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829106.911277:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829106.911281:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564356736:192.168.203.139@tcp:38:kworker.0 00000100:00080000:1.0:1787829106.911285:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4046 00000100:00080000:1.0:1787829106.911293:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829106.911298:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829106.911304:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829106.911320:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564356992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829106.911322:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564356864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829106.911363:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564356992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829107.166984:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829107.166984:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829107.167100:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829107.934960:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829108.190905:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829108.190921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829108.190976:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829108.958929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829109.214920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829109.214920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829109.214995:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829109.982926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:0.0:1787829109.983172:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:0.0:1787829109.983223:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829109.983262:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787829109.983319:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787829109.983335:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787829109.983356:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:3.0:1787829109.983357:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829106/real 1787829109] req@ffff976889efed00 x1874670564356608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787829109.983369:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:3.0:1787829109.983398:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564356608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829109.983413:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829109.983419:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829109.983425:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564356608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829109.983435:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829106/real 1787829109] req@ffff976889effc00 x1874670564356736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829109.983450:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564356736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829109.983460:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829109.983464:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829109.983468:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564356736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829109.983475:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829106/real 1787829109] req@ffff976889efc3c0 x1874670564356480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829109.983490:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564356480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829109.983498:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829109.983504:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829109.983508:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564356480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829109.983568:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829106/real 1787829109] req@ffff976889efe940 x1874670564356992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829109.983585:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564356992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829109.983594:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829109.983599:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829109.983607:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564356992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829109.983617:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829106/real 1787829109] req@ffff976889eff0c0 x1874670564356864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829109.983631:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564356864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829109.983641:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829109.983644:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829109.983648:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564356864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829110.238889:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829110.238902:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829110.238978:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829111.006902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829111.262960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829111.262962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829111.263020:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829112.030934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829112.031004:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829112.031015:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829112.031018:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829112.031026:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4051 00000100:00080000:2.0:1787829112.031035:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829112.031041:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829112.031070:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564357120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829112.031092:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829112.031095:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829112.031096:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829112.031099:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4051 00000100:00080000:2.0:1787829112.031102:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829112.031104:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829112.031112:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829112.031122:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564357248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829112.031131:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829112.031136:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829112.031138:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829112.031142:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4051 00000100:00080000:2.0:1787829112.031152:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829112.031152:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564357120:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829112.031164:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829112.031172:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829112.031187:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564357376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829112.031208:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829112.031211:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829112.031214:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829112.031219:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4051 00000100:00080000:2.0:1787829112.031223:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829112.031228:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829112.031235:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829112.031247:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564357504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829112.031256:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829112.031260:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829112.031261:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829112.031266:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4051 00000100:00080000:2.0:1787829112.031271:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829112.031279:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829112.031288:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829112.031296:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829112.031313:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564357632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829112.031374:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829112/real 1787829112] req@ffff976889cf92c0 x1874670564357120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829112.031404:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564357120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829112.031418:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829112.031424:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829112.031428:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564357120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829112.031437:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564357248:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829112.031467:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829112.031483:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829112/real 1787829112] req@ffff976889cfa1c0 x1874670564357248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829112.031495:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564357248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829112.031504:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829112.031508:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829112.031525:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564357248:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829112.031562:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564357376:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829112.031585:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829112.031596:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829112/real 1787829112] req@ffff976889cfad00 x1874670564357376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829112.031609:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564357376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829112.031619:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829112.031623:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829112.031626:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564357376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829112.031634:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564357504:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829112.031656:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829112.031668:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829112/real 1787829112] req@ffff976889cfb480 x1874670564357504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829112.031682:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564357504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829112.031691:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829112.031694:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829112.031698:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564357504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829112.031706:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564357632:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829112.031727:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829112.031861:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829112/real 1787829112] req@ffff976889cfa940 x1874670564357632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829112.031879:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564357632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829112.031888:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829112.031891:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829112.031895:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564357632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829112.286931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829112.286931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829112.287016:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829113.054911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829113.310966:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829113.310995:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829113.311030:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829114.078919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829114.334950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829114.334951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829114.335008:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829115.102923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829115.358893:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829115.358949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829115.359029:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829116.126997:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829116.382983:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829116.382984:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829116.383054:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829117.150921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829117.150936:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829117.150944:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829117.150949:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829117.150969:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4057 00000100:00080000:2.0:1787829117.150979:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829117.150985:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829117.151019:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564357760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829117.151057:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829117.151060:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829117.151061:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829117.151064:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4057 00000100:00080000:2.0:1787829117.151066:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829117.151069:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829117.151078:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829117.151092:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564357888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829117.151105:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829117.151107:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564357760:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829117.151126:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829117.151133:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829117.151143:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4057 00000100:00080000:2.0:1787829117.151146:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829117.151151:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829117.151156:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829117.151167:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564358016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829117.151175:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829117.151179:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829117.151182:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829117.151204:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4057 00000100:00080000:2.0:1787829117.151226:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829117.151233:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829117.151240:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787829117.151250:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829117.151256:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564358144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829117.151282:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829117.151286:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829117.151288:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829117.151294:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4057 00000100:00080000:2.0:1787829117.151300:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829117.151305:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829117.151312:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829117.151339:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564358272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829117.151350:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829117/real 1787829117] req@ffff976889cf8f00 x1874670564357760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829117.151378:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564357760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829117.151406:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829117.151411:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829117.151415:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564357760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829117.151447:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564357888:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829117.151471:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829117.151504:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829117/real 1787829117] req@ffff976889cf8000 x1874670564357888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829117.151531:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564357888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829117.151537:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829117.151539:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829117.151542:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564357888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829117.151547:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564358016:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829117.151563:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829117.151570:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829117/real 1787829117] req@ffff976889cfb0c0 x1874670564358016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829117.151577:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564358016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829117.151583:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829117.151586:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829117.151588:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564358016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829117.151592:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564358144:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829117.151603:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829117.151610:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829117/real 1787829117] req@ffff976889cf9e00 x1874670564358144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829117.151617:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564358144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829117.151622:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829117.151624:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829117.151626:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564358144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829117.151630:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564358272:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829117.151641:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829117.151646:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829117/real 1787829117] req@ffff976889cfa580 x1874670564358272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829117.151653:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564358272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829117.151658:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829117.151660:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829117.151662:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564358272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829117.406881:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829117.406932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829117.406987:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829118.174930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829118.430963:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829118.431000:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829118.431021:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829119.198909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829119.454953:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829119.454958:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829119.455038:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829120.222961:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829120.478970:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829120.478970:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829120.479067:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829121.246932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829121.502948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829121.502957:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829121.503024:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829122.270949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829122.270980:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829122.270987:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829122.270992:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829122.271004:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4062 00000100:00080000:1.0:1787829122.271019:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829122.271028:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829122.271085:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564358400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829122.271110:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829122.271113:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829122.271119:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829122.271124:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4062 00000100:00080000:1.0:1787829122.271128:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829122.271132:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829122.271143:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564358400:192.168.203.139@tcp:250:kworker.0 00000008:00100000:1.0:1787829122.271144:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829122.271164:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564358528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829122.271177:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829122.271182:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829122.271185:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829122.271189:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4062 00000100:00080000:1.0:1787829122.271194:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829122.271198:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829122.271204:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829122.271216:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564358656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829122.271227:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787829122.271230:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787829122.271232:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829122.271237:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829122.271251:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4062 00000100:00080000:1.0:1787829122.271258:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829122.271263:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829122.271270:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829122.271287:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564358784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829122.271295:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829122/real 1787829122] req@ffff976889efcb40 x1874670564358400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829122.271302:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829122.271313:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829122.271319:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829122.271324:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4062 00000100:00080000:1.0:1787829122.271328:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829122.271331:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564358400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829122.271335:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829122.271345:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787829122.271349:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829122.271355:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829122.271358:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564358912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829122.271362:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564358400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829122.271395:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564358528:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829122.271421:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829122.271434:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829122/real 1787829122] req@ffff976889efe1c0 x1874670564358528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829122.271461:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564358528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829122.271472:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829122.271475:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829122.271479:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564358528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829122.271486:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564358656:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829122.271506:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829122.271516:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829122/real 1787829122] req@ffff976889efe580 x1874670564358656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829122.271529:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564358656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829122.271545:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829122.271549:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829122.271553:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564358656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829122.271558:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564358784:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829122.271574:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829122.271583:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829122/real 1787829122] req@ffff976889eff480 x1874670564358784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829122.271594:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564358784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829122.271603:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829122.271607:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829122.271610:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564358784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829122.271616:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564358912:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829122.271636:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829122.271648:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829122/real 1787829122] req@ffff976889efc780 x1874670564358912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829122.271660:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564358912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829122.271668:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829122.271671:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829122.271675:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564358912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829122.526940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829122.526942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829122.527018:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829123.294916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829123.550955:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829123.550961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829123.550993:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829124.318941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829124.574949:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829124.574964:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829124.575103:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829125.342938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829125.598927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829125.598927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829125.598987:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829126.366945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829126.622960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829126.622968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829126.623038:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829127.390930:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829127.390936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829127.390940:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829127.390953:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829127.390964:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4067 00000100:00080000:1.0:1787829127.390979:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829127.390989:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829127.391061:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564359040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829127.391096:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829127.391102:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829127.391104:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829127.391108:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4067 00000100:00080000:1.0:1787829127.391113:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829127.391134:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829127.391147:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829127.391153:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564359040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829127.391166:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564359168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829127.391187:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829127.391190:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829127.391192:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829127.391197:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4067 00000100:00080000:1.0:1787829127.391202:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829127.391207:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829127.391213:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829127.391228:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564359296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829127.391238:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787829127.391246:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829127.391249:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829127.391250:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829127.391255:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4067 00000100:00080000:1.0:1787829127.391258:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829127.391262:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829127.391267:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829127.391278:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564359424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829127.391285:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829127.391287:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829127.391288:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829127.391291:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4067 00000100:00080000:1.0:1787829127.391294:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829127.391298:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829127.391302:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829127.391307:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829127/real 1787829127] req@ffff976889efc780 x1874670564359040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829127.391312:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564359552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829127.391335:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564359040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829127.391345:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829127.391349:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829127.391352:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564359040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829127.391376:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564359168:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829127.391395:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829127.391421:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829127/real 1787829127] req@ffff976889eff480 x1874670564359168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829127.391431:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564359168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829127.391437:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829127.391440:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829127.391443:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564359168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829127.391447:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564359296:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829127.391463:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829127.391471:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829127/real 1787829127] req@ffff976889efe580 x1874670564359296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829127.391478:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564359296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829127.391484:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829127.391486:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829127.391488:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564359296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829127.391492:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564359424:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829127.391502:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829127.391508:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829127/real 1787829127] req@ffff976889efe1c0 x1874670564359424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829127.391515:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564359424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829127.391520:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829127.391522:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829127.391524:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564359424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829127.391527:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564359552:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829127.391537:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829127.391543:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829127/real 1787829127] req@ffff976889efcb40 x1874670564359552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829127.391551:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564359552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829127.391556:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829127.391558:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829127.391560:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564359552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829127.646903:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829127.646908:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829127.646996:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829128.414905:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829128.670932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829128.670947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829128.670962:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829129.438957:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829129.694909:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829129.694909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829129.694976:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829130.462954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829130.718967:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829130.718974:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829130.719059:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829131.486967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829131.742973:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829131.742974:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829131.743043:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829132.510918:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829132.510929:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1787829132.510930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829132.510934:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829132.510948:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4072 00000100:00080000:1.0:1787829132.510960:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829132.510969:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829132.511007:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564359680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829132.511042:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829132.511046:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829132.511049:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829132.511055:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4072 00000100:00080000:1.0:1787829132.511060:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829132.511066:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829132.511079:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829132.511083:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564359680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829132.511101:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564359808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829132.511116:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829132.511132:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829132.511135:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829132.511144:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4072 00000100:00080000:1.0:1787829132.511153:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829132.511159:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829132.511167:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829132.511172:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829132.511187:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564359936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829132.511201:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829132.511205:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829132.511208:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829132.511214:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4072 00000100:00100000:2.0:1787829132.511217:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829132/real 1787829132] req@ffff976889eff0c0 x1874670564359680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829132.511221:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829132.511232:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829132.511240:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829132.511246:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564359680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829132.511259:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564360064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829132.511262:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829132.511272:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:1.0:1787829132.511281:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829132.511282:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564359680:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829132.511292:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829132.511296:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829132.511303:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4072 00000100:00080000:1.0:1787829132.511308:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829132.511314:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829132.511320:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564359808:192.168.203.139@tcp:38:kworker.0 00000008:00100000:1.0:1787829132.511323:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829132.511343:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564360192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829132.511357:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829132.511389:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829132/real 1787829132] req@ffff976889efe940 x1874670564359808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829132.511404:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564359808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829132.511411:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829132.511413:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829132.511416:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564359808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829132.511421:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564359936:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829132.511440:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829132.511452:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829132/real 1787829132] req@ffff976889efc3c0 x1874670564359936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829132.511465:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564359936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829132.511474:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829132.511478:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829132.511482:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564359936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829132.511489:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564360064:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829132.511511:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829132.511523:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829132/real 1787829132] req@ffff976889effc00 x1874670564360064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829132.511535:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564360064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829132.511550:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829132.511553:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829132.511557:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564360064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829132.511575:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564360192:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829132.511597:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829132.511613:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829132/real 1787829132] req@ffff976889efed00 x1874670564360192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829132.511629:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564360192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829132.511642:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829132.511646:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829132.511649:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564360192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829132.766937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829132.766947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829132.766985:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829133.534935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829133.790941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829133.790941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829133.790979:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829134.558963:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829134.814916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829134.814965:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829134.815053:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829135.582939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829135.838960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829135.838960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829135.839024:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829136.606904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829136.862886:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829136.862887:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829136.862962:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829137.630962:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829137.630962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829137.630971:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829137.630974:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829137.630984:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4077 00000100:00080000:1.0:1787829137.630994:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829137.631001:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829137.631035:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564360320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829137.631056:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829137.631058:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829137.631059:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829137.631063:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4077 00000100:00080000:1.0:1787829137.631065:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829137.631067:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829137.631075:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829137.631084:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564360448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829137.631092:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829137.631095:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829137.631096:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829137.631098:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4077 00000100:00080000:1.0:1787829137.631100:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829137.631104:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829137.631106:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564360320:192.168.203.139@tcp:250:kworker.0 00000008:00100000:1.0:1787829137.631107:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829137.631117:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564360576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829137.631125:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829137.631128:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829137.631131:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829137.631135:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4077 00000100:00080000:1.0:1787829137.631138:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829137.631142:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829137.631148:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829137.631162:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564360704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829137.631178:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787829137.631181:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829137.631186:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829137.631188:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829137.631192:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4077 00000100:00080000:1.0:1787829137.631201:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829137.631205:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829137.631212:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829137.631225:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829137/real 1787829137] req@ffff976889efed00 x1874670564360320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829137.631228:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564360832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829137.631268:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564360320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829137.631281:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829137.631285:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829137.631289:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564360320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829137.631297:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564360448:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829137.631320:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829137.631350:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829137/real 1787829137] req@ffff976889effc00 x1874670564360448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829137.631365:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564360448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829137.631373:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829137.631377:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829137.631380:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564360448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829137.631411:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564360576:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829137.631434:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829137.631447:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829137/real 1787829137] req@ffff976889efc3c0 x1874670564360576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829137.631457:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564360576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829137.631463:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829137.631465:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829137.631468:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564360576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829137.631472:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564360704:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829137.631487:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829137.631494:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829137/real 1787829137] req@ffff976889efe940 x1874670564360704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829137.631501:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564360704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829137.631506:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829137.631508:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829137.631510:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564360704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829137.631513:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564360832:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829137.631523:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829137.631529:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829137/real 1787829137] req@ffff976889eff0c0 x1874670564360832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829137.631535:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564360832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829137.631540:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829137.631542:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829137.631544:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564360832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829137.886945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829137.886945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829137.887006:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829138.654914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829138.910928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829138.910929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829138.910957:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829139.678946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829139.934936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829139.934937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829139.934991:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829140.702920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829140.958918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829140.958918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829140.958983:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829141.726927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829141.982922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829141.982922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829141.982980:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829142.750889:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829142.750899:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829142.750903:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829142.750905:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829142.750912:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4082 00000100:00080000:1.0:1787829142.750923:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829142.750931:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829142.750972:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564360960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829142.751000:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829142.751004:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829142.751006:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829142.751011:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4082 00000100:00080000:1.0:1787829142.751015:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829142.751019:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829142.751030:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829142.751043:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564361088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829142.751050:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829142.751053:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829142.751054:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829142.751057:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4082 00000100:00080000:1.0:1787829142.751059:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829142.751062:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829142.751066:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829142.751073:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564361216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829142.751080:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829142.751085:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564360960:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829142.751086:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829142.751089:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829142.751097:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4082 00000100:00080000:1.0:1787829142.751100:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829142.751105:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829142.751110:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829142.751124:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564361344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829142.751135:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829142.751139:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829142.751142:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829142.751148:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4082 00000800:00000100:2.0:1787829142.751149:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787829142.751156:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829142.751160:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829142.751164:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829142.751174:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564361472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829142.751193:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829142/real 1787829142] req@ffff976889efcb40 x1874670564360960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829142.751207:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564360960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829142.751215:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829142.751217:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829142.751221:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564360960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829142.751226:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564361088:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829142.751239:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829142.751247:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829142/real 1787829142] req@ffff976889efe1c0 x1874670564361088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829142.751255:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564361088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829142.751260:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829142.751263:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829142.751265:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564361088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829142.751282:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564361216:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829142.751293:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829142.751318:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829142/real 1787829142] req@ffff976889efe580 x1874670564361216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829142.751326:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564361216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829142.751331:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829142.751333:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829142.751336:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564361216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829142.751339:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564361344:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829142.751352:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829142.751358:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829142/real 1787829142] req@ffff976889eff480 x1874670564361344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829142.751365:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564361344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829142.751370:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829142.751372:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829142.751374:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564361344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829142.751377:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564361472:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829142.751387:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829142.751392:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829142/real 1787829142] req@ffff976889efc780 x1874670564361472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829142.751407:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564361472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829142.751413:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829142.751415:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829142.751417:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564361472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829143.006904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829143.006928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829143.006993:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829143.774909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829144.030908:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829144.030913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829144.030947:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829144.798899:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829145.054892:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829145.054945:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829145.054976:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829145.822883:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829146.078921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829146.078922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829146.078949:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829146.846966:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829147.102887:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829147.102892:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829147.102955:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829147.870916:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829147.870925:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829147.870929:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829147.870933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829147.870939:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4087 00000100:00080000:2.0:1787829147.870948:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829147.870955:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829147.870987:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564361600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829147.871013:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829147.871016:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829147.871017:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829147.871020:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4087 00000100:00080000:2.0:1787829147.871022:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829147.871025:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829147.871032:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829147.871042:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564361728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829147.871052:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829147.871055:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829147.871056:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829147.871061:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564361600:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829147.871062:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4087 00000100:00080000:2.0:1787829147.871068:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829147.871071:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829147.871074:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829147.871082:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564361856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829147.871088:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829147.871090:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829147.871092:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829147.871095:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4087 00000100:00080000:2.0:1787829147.871097:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829147.871100:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829147.871103:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829147.871111:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564361984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829147.871117:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829147.871120:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829147.871121:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829147.871123:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4087 00000100:00080000:2.0:1787829147.871126:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829147.871129:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829147.871135:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787829147.871135:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829147.871145:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564362112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829147.871192:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829147/real 1787829147] req@ffff976889cfbc00 x1874670564361600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829147.871208:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564361600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829147.871216:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829147.871218:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829147.871221:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564361600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829147.871241:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564361728:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829147.871258:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829147.871285:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829147/real 1787829147] req@ffff976889cfa580 x1874670564361728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829147.871294:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564361728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829147.871301:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829147.871303:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829147.871305:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564361728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829147.871310:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564361856:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829147.871324:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829147.871331:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829147/real 1787829147] req@ffff976889cf9e00 x1874670564361856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829147.871339:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564361856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829147.871348:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829147.871350:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829147.871353:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564361856:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829147.871356:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564361984:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829147.871366:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829147.871372:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829147/real 1787829147] req@ffff976889cfb0c0 x1874670564361984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829147.871379:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564361984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829147.871384:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829147.871386:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829147.871388:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564361984:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829147.871391:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564362112:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829147.871401:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829147.871406:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829147/real 1787829147] req@ffff976889cf8000 x1874670564362112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829147.871413:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564362112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829147.871418:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829147.871420:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829147.871423:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564362112:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829148.126920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829148.126920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829148.126954:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829148.894960:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829149.150879:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829149.150906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829149.150981:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829149.918873:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829150.174907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829150.174938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829150.175014:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829150.942997:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829151.198984:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829151.198991:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829151.199205:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829151.966906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829152.222935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829152.222943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829152.223005:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829152.990906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829152.990926:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829152.990932:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829152.990936:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829152.990944:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4092 00000100:00080000:1.0:1787829152.990953:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829152.990958:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829152.990993:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564362240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829152.991015:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829152.991018:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829152.991019:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829152.991022:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4092 00000100:00080000:1.0:1787829152.991025:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829152.991028:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829152.991037:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829152.991046:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564362368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829152.991051:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564362240:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829152.991061:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829152.991064:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829152.991065:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829152.991068:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4092 00000100:00080000:1.0:1787829152.991070:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829152.991072:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829152.991075:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829152.991084:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564362496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829152.991090:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829152.991093:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829152.991094:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829152.991096:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4092 00000100:00080000:1.0:1787829152.991100:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829152.991102:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829152.991106:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829152.991114:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564362624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829152.991120:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787829152.991127:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829152.991130:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829152.991132:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829152.991136:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4092 00000100:00080000:1.0:1787829152.991141:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829152.991143:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829152.991146:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829152.991153:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564362752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829152.991160:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829152/real 1787829152] req@ffff976889efc780 x1874670564362240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829152.991175:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564362240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829152.991184:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829152.991187:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829152.991190:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564362240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829152.991210:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564362368:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829152.991240:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829152.991253:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829152/real 1787829152] req@ffff976889eff480 x1874670564362368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829152.991266:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564362368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829152.991274:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829152.991276:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829152.991278:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564362368:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829152.991283:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564362496:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829152.991298:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829152.991305:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829152/real 1787829152] req@ffff976889efe580 x1874670564362496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829152.991313:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564362496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829152.991318:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829152.991320:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829152.991322:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564362496:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829152.991325:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564362624:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829152.991335:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829152.991341:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829152/real 1787829152] req@ffff976889efe1c0 x1874670564362624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829152.991348:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564362624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829152.991353:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829152.991355:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829152.991357:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564362624:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829152.991360:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564362752:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829152.991369:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829152.991375:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829152/real 1787829152] req@ffff976889efcb40 x1874670564362752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829152.991387:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564362752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829152.991392:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829152.991393:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829152.991395:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564362752:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829153.246967:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829153.247071:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829153.247096:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829154.014896:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829154.270913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787829154.270947:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829154.271057:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829155.038900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829155.294948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829155.294957:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829155.295001:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829156.062951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829156.318893:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829156.318948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829156.318955:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829157.086930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829157.342908:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829157.342920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829157.342952:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829158.110946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829158.110946:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829158.110958:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829158.110964:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829158.110975:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4098 00000100:00080000:1.0:1787829158.110989:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829158.110999:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829158.111044:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564362880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829158.111074:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829158.111078:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829158.111080:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829158.111085:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4098 00000100:00080000:1.0:1787829158.111089:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829158.111092:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829158.111103:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829158.111117:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564363008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829158.111129:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829158.111133:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564362880:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829158.111134:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829158.111139:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829158.111145:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4098 00000100:00080000:1.0:1787829158.111148:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829158.111152:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829158.111158:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829158.111168:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564363136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829158.111179:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829158.111182:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829158.111185:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829158.111200:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4098 00000100:00080000:1.0:1787829158.111204:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829158.111208:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829158.111214:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829158.111226:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564363264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829158.111228:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787829158.111240:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829158.111244:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829158.111247:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829158.111253:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4098 00000100:00080000:1.0:1787829158.111256:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829158.111260:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829158.111265:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829158.111279:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829158/real 1787829158] req@ffff976889eff0c0 x1874670564362880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829158.111280:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564363392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829158.111310:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564362880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829158.111322:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829158.111325:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829158.111328:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564362880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829158.111353:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564363008:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829158.111370:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829158.111378:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829158/real 1787829158] req@ffff976889efe940 x1874670564363008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829158.111386:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564363008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829158.111393:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829158.111395:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829158.111398:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564363008:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829158.111402:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564363136:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829158.111412:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829158.111417:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829158/real 1787829158] req@ffff976889efc3c0 x1874670564363136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829158.111425:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564363136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829158.111431:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829158.111432:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829158.111434:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564363136:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829158.111438:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564363264:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829158.111449:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829158.111455:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829158/real 1787829158] req@ffff976889effc00 x1874670564363264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829158.111463:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564363264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829158.111468:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829158.111470:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829158.111472:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564363264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829158.111476:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564363392:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829158.111486:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829158.111491:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829158/real 1787829158] req@ffff976889efed00 x1874670564363392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829158.111497:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564363392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829158.111503:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829158.111504:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829158.111506:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564363392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829158.366908:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829158.366946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829158.366993:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829159.134926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829159.390911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829159.390929:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829159.390958:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829160.158968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829160.414946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829160.415010:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829160.415068:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829161.182917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829161.438918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829161.438923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829161.438948:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829162.206978:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829162.462900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829162.462974:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829162.462976:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00080000:0.0:1787829163.230936:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829163.230943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829163.230956:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829163.230962:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829163.230975:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4103 00000100:00080000:0.0:1787829163.230989:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829163.231006:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829163.231058:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564363520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829163.231094:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829163.231099:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829163.231102:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829163.231107:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4103 00000100:00080000:0.0:1787829163.231111:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829163.231117:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829163.231132:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829163.231150:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564363648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829163.231176:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829163.231184:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787829163.231186:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564363520:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829163.231187:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829163.231204:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4103 00000100:00080000:0.0:1787829163.231210:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829163.231216:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829163.231223:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829163.231241:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564363776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829163.231254:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829163.231259:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829163.231262:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787829163.231270:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829163.231276:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4103 00000100:00080000:0.0:1787829163.231283:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829163.231291:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829163.231298:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829163.231316:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564363904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829163.231325:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829163/real 1787829163] req@ffff976882aed680 x1874670564363520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829163.231344:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829163.231350:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829163.231353:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829163.231358:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564363520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829163.231366:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4103 00000100:00080000:3.0:1787829163.231375:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829163.231376:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829163.231384:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:0.0:1787829163.231389:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829163.231393:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564363520:192.168.203.139@tcp:250:kworker.0 00000008:00100000:0.0:1787829163.231405:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829163.231423:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564363648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829163.231427:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564364032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829163.231447:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829163.231462:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829163/real 1787829163] req@ffff976882aee940 x1874670564363648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829163.231477:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564363648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829163.231487:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829163.231491:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829163.231495:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564363648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829163.231502:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564363776:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829163.231523:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829163.231535:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829163/real 1787829163] req@ffff976882aef0c0 x1874670564363776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829163.231548:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564363776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829163.231554:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829163.231557:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829163.231559:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564363776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829163.231563:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564363904:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829163.231579:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829163.231586:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829163/real 1787829163] req@ffff976882aeed00 x1874670564363904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829163.231594:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564363904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829163.231600:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829163.231602:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829163.231604:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564363904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829163.231618:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564364032:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829163.231629:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829163.231635:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829163/real 1787829163] req@ffff976882aee580 x1874670564364032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829163.231643:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564364032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829163.231648:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829163.231651:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829163.231653:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564364032:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829163.486942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829163.486974:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829163.486991:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829164.254942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829164.510911:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829164.510938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829164.510977:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829165.278937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829165.534919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829165.534944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829165.535019:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829166.302933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829166.558902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829166.558935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829166.558993:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829167.326953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829167.582966:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829167.582981:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829167.583095:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829168.350993:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829168.351016:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829168.351029:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829168.351045:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829168.351058:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4108 00000100:00080000:0.0:1787829168.351073:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829168.351082:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829168.351154:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564364160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829168.351207:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829168.351210:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829168.351211:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829168.351215:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4108 00000100:00080000:0.0:1787829168.351218:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829168.351221:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829168.351232:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829168.351245:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564364288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829168.351253:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829168.351257:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829168.351260:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829168.351264:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4108 00000100:00080000:0.0:1787829168.351266:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829168.351268:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829168.351271:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829168.351278:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564364416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829168.351285:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829168.351298:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829168.351299:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829168.351303:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4108 00000100:00080000:0.0:1787829168.351306:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829168.351310:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829168.351315:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829168.351331:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564364544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829168.351333:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564364160:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829168.351342:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829168.351355:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829168.351373:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829168.351378:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4108 00000100:00080000:0.0:1787829168.351384:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829168.351391:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829168.351397:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829168.351416:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564364672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829168.351476:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829168.351560:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829168/real 1787829168] req@ffff976882aec3c0 x1874670564364160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829168.351586:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564364160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829168.351609:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829168.351614:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829168.351619:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564364160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829168.351630:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564364288:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829168.351661:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829168.351685:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829168/real 1787829168] req@ffff976882aee580 x1874670564364288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829168.351702:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564364288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829168.351715:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829168.351721:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829168.351725:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564364288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829168.351732:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564364416:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829168.351750:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829168.351760:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829168/real 1787829168] req@ffff976882aeed00 x1874670564364416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829168.351773:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564364416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829168.351781:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829168.351921:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829168.351928:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564364416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829168.351977:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564364544:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829168.352211:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829168.352282:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829168/real 1787829168] req@ffff976882aef0c0 x1874670564364544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829168.352303:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564364544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829168.352313:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829168.352318:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829168.352323:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564364544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829168.352331:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564364672:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829168.352364:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829168.352376:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829168/real 1787829168] req@ffff976882aee940 x1874670564364672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829168.352387:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564364672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829168.352395:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829168.352398:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829168.352401:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564364672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829168.607007:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829168.607015:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829168.607101:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829169.374933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829169.630958:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829169.630993:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829169.631028:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829170.398968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829170.654965:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829170.654966:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829170.655181:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829171.422914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829171.678906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829171.678933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829171.679048:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829172.446981:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829172.703027:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829172.703054:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829172.703085:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829173.470932:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829173.470933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829173.470942:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829173.470947:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829173.470955:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4113 00000100:00080000:0.0:1787829173.470963:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829173.470969:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829173.471003:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564364800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829173.471025:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829173.471028:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829173.471029:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829173.471032:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4113 00000100:00080000:0.0:1787829173.471034:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829173.471037:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829173.471046:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829173.471059:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564364928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829173.471061:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564364800:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829173.471076:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829173.471079:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829173.471080:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829173.471096:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4113 00000100:00080000:0.0:1787829173.471099:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829173.471102:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829173.471106:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829173.471115:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564365056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829173.471122:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829173.471124:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829173.471125:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829173.471128:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4113 00000100:00080000:0.0:1787829173.471131:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829173.471133:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829173.471136:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829173.471142:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564365184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829173.471149:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829173.471151:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829173.471152:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829173.471154:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4113 00000100:00080000:0.0:1787829173.471156:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829173.471158:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829173.471161:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829173.471167:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564365312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829173.471211:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564364928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829173.471247:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564365056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829173.471274:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564365184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829173.471303:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564365312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829173.726946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829173.727017:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829173.727044:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829174.494996:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829174.750961:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829174.750961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829174.751090:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829175.519050:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829175.774930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829175.775013:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829175.775089:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829176.543114:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:0.0:1787829176.543256:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:0.0:1787829176.543327:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829176.543351:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787829176.543413:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787829176.543426:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787829176.543441:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787829176.543460:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:3.0:1787829176.543496:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829173/real 1787829176] req@ffff976882aee940 x1874670564364928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829176.543534:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564364928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829176.543547:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829176.543552:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829176.543557:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564364928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829176.543566:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829173/real 1787829176] req@ffff976882aef0c0 x1874670564365056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829176.543579:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564365056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829176.543588:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829176.543592:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829176.543595:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564365056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829176.543603:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829173/real 1787829176] req@ffff976882aeed00 x1874670564365184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829176.543616:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564365184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829176.543625:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829176.543631:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829176.543634:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564365184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829176.543641:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829173/real 1787829176] req@ffff976882aee580 x1874670564365312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829176.543655:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564365312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829176.543663:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829176.543667:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829176.543671:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564365312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829176.543679:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829173/real 1787829176] req@ffff976882aed680 x1874670564364800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829176.543710:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564364800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829176.543720:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829176.543723:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829176.543727:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564364800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829176.798946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829176.799033:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829176.799042:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829177.567001:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829177.822934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829177.822934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829177.823026:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829178.590938:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829178.590948:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829178.590953:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829178.590963:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4118 00000100:00100000:1.0:1787829178.590971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829178.590976:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829178.590988:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829178.591029:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564365440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829178.591068:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829178.591072:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829178.591073:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829178.591077:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4118 00000100:00080000:2.0:1787829178.591080:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829178.591083:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829178.591097:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829178.591113:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564365568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829178.591127:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829178.591128:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564365440:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829178.591135:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829178.591139:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829178.591146:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4118 00000100:00080000:2.0:1787829178.591150:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829178.591155:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829178.591161:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829178.591176:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564365696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829178.591188:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829178.591192:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829178.591194:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829178.591199:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4118 00000100:00080000:2.0:1787829178.591204:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829178.591215:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829178.591222:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787829178.591224:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829178.591239:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564365824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829178.591253:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829178.591257:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829178.591259:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829178.591263:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4118 00000100:00080000:2.0:1787829178.591268:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829178.591272:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829178.591278:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829178.591286:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829178/real 1787829178] req@ffff976889cf8f00 x1874670564365440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829178.591295:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564365952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829178.591323:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564365440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829178.591339:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829178.591345:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829178.591350:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564365440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829178.591386:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564365568:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829178.591416:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829178.591431:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829178/real 1787829178] req@ffff976889cfa940 x1874670564365568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829178.591444:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564365568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829178.591454:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829178.591458:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829178.591461:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564365568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829178.591468:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564365696:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829178.591490:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829178.591501:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829178/real 1787829178] req@ffff976889cfb480 x1874670564365696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829178.591513:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564365696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829178.591522:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829178.591525:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829178.591528:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564365696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829178.591535:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564365824:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829178.591557:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829178.591570:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829178/real 1787829178] req@ffff976889cfad00 x1874670564365824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829178.591582:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564365824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829178.591590:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829178.591593:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829178.591597:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564365824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829178.591602:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564365952:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829178.591623:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829178.591634:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829178/real 1787829178] req@ffff976889cfa1c0 x1874670564365952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829178.591646:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564365952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829178.591655:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829178.591659:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829178.591663:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564365952:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829178.846929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829178.846930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829178.846989:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829179.615076:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829179.870938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829179.870948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829179.871117:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829180.638984:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829180.894951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829180.894954:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829180.895025:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829181.662906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829181.918927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829181.918927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829181.918955:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829182.686963:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829182.942957:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829182.942963:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829182.943081:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829183.710914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829183.710920:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829183.710928:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829183.710933:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829183.710943:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4123 00000100:00080000:2.0:1787829183.710956:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829183.710965:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829183.711008:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564366080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829183.711067:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829183.711070:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829183.711072:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829183.711076:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4123 00000100:00080000:2.0:1787829183.711078:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829183.711082:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829183.711092:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829183.711124:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564366208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829183.711134:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564366080:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829183.711136:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829183.711144:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829183.711148:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829183.711153:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4123 00000100:00080000:2.0:1787829183.711162:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829183.711165:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829183.711172:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829183.711182:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564366336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829183.711189:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829183.711192:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829183.711193:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829183.711196:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4123 00000100:00080000:2.0:1787829183.711217:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829183.711222:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829183.711228:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787829183.711239:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829183.711244:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564366464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829183.711273:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829183.711277:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829183.711279:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829183.711284:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4123 00000100:00080000:2.0:1787829183.711288:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829183.711292:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829183.711296:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829183.711307:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564366592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829183.711317:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829183/real 1787829183] req@ffff976889cf92c0 x1874670564366080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829183.711359:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564366080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829183.711369:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829183.711373:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829183.711376:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564366080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829183.711402:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564366208:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829183.711424:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829183.711434:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829183/real 1787829183] req@ffff976889cfa1c0 x1874670564366208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829183.711444:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564366208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829183.711451:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829183.711454:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829183.711456:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564366208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829183.711460:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564366336:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829183.711472:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829183.711478:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829183/real 1787829183] req@ffff976889cfad00 x1874670564366336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829183.711493:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564366336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829183.711498:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829183.711500:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829183.711502:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564366336:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829183.711506:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564366464:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829183.711518:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829183.711525:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829183/real 1787829183] req@ffff976889cfb480 x1874670564366464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829183.711544:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564366464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829183.711550:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829183.711552:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829183.711555:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564366464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829183.711558:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564366592:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829183.711587:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829183.711594:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829183/real 1787829183] req@ffff976889cfa940 x1874670564366592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829183.711601:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564366592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829183.711611:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829183.711613:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829183.711615:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564366592:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829183.966934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829183.966959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829183.967023:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829184.734919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829184.990939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829184.990966:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829184.991033:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829185.758982:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829186.014887:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829186.014897:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829186.014942:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829186.782900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829187.038891:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829187.038901:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829187.039051:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829187.806945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829188.063016:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829188.063017:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829188.063063:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829188.830904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829188.830925:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829188.830934:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829188.830939:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829188.830951:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4128 00000100:00080000:0.0:1787829188.830966:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829188.830974:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829188.831035:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564366720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829188.831067:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829188.831070:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829188.831073:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829188.831077:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4128 00000100:00080000:0.0:1787829188.831081:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829188.831085:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829188.831097:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829188.831114:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564366848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829188.831123:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564366720:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829188.831129:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829188.831134:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829188.831136:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829188.831141:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4128 00000100:00080000:0.0:1787829188.831145:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829188.831149:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829188.831154:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829188.831168:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564366976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829188.831178:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829188.831181:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829188.831183:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829188.831187:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4128 00000100:00080000:0.0:1787829188.831192:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829188.831195:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829188.831202:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787829188.831205:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829188.831217:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564367104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829188.831228:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829188.831232:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829188.831234:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829188.831239:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4128 00000100:00080000:0.0:1787829188.831243:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829188.831248:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829188.831252:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829188/real 1787829188] req@ffff976882aec3c0 x1874670564366720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829188.831256:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829188.831275:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564367232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829188.831283:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564366720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829188.831297:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829188.831301:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829188.831305:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564366720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829188.831328:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564366848:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829188.831345:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829188.831354:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829188/real 1787829188] req@ffff976882aed680 x1874670564366848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829188.831363:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564366848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829188.831373:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829188.831376:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829188.831378:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564366848:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829188.831382:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564366976:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829188.831392:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829188.831398:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829188/real 1787829188] req@ffff976882aee580 x1874670564366976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829188.831405:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564366976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829188.831410:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829188.831412:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829188.831414:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564366976:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829188.831417:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564367104:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829188.831429:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829188.831435:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829188/real 1787829188] req@ffff976882aeed00 x1874670564367104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829188.831442:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564367104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829188.831447:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829188.831449:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829188.831452:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564367104:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829188.831455:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564367232:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829188.831465:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829188.831470:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829188/real 1787829188] req@ffff976882aef0c0 x1874670564367232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829188.831477:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564367232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829188.831483:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829188.831485:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829188.831487:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564367232:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829189.086891:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829189.086904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829189.087084:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829189.854922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829190.110939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829190.110964:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829190.110994:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829190.878900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829191.134990:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829191.135007:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829191.135083:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829191.903094:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829192.159009:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829192.159012:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829192.159053:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829192.926973:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829193.182899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829193.182906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829193.183055:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829193.950967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829193.951001:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829193.951012:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829193.951018:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829193.951031:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4133 00000100:00080000:0.0:1787829193.951043:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829193.951052:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829193.951096:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564367360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829193.951126:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829193.951132:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829193.951135:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829193.951141:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4133 00000100:00080000:0.0:1787829193.951146:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829193.951152:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829193.951178:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829193.951199:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564367488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829193.951215:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829193.951215:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564367360:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829193.951220:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829193.951223:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829193.951227:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4133 00000100:00080000:0.0:1787829193.951232:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829193.951237:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829193.951244:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829193.951257:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564367616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829193.951269:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829193.951273:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829193.951276:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829193.951281:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4133 00000100:00080000:0.0:1787829193.951285:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829193.951289:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:3.0:1787829193.951293:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:0.0:1787829193.951296:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829193.951309:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564367744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829193.951321:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829193.951325:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829193.951328:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829193.951332:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4133 00000100:00080000:0.0:1787829193.951337:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829193.951341:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829193.951343:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829193/real 1787829193] req@ffff976882aee940 x1874670564367360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829193.951348:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829193.951360:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564367872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829193.951368:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564367360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829193.951396:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829193.951403:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829193.951418:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564367360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829193.951449:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564367488:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829193.951477:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829193.951490:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829193/real 1787829193] req@ffff976882aef0c0 x1874670564367488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829193.951502:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564367488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829193.951511:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829193.951514:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829193.951517:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564367488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829193.951522:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564367616:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829193.951538:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829193.951546:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829193/real 1787829193] req@ffff976882aeed00 x1874670564367616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829193.951557:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564367616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829193.951564:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829193.951567:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829193.951569:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564367616:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829193.951574:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564367744:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829193.951588:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829193.951596:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829193/real 1787829193] req@ffff976882aee580 x1874670564367744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829193.951621:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564367744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829193.951628:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829193.951631:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829193.951633:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564367744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829193.951638:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564367872:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829193.951652:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829193.951660:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829193/real 1787829193] req@ffff976882aed680 x1874670564367872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829193.951670:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564367872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829193.951677:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829193.951680:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829193.951682:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564367872:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829194.207040:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829194.207041:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829194.207092:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829194.975036:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829195.230957:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829195.230966:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829195.231018:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829195.998895:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829196.255045:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829196.255047:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829196.255103:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829197.023018:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829197.278938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829197.278979:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829197.279079:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829198.046910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829198.302935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829198.302939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829198.303071:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829199.070937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829199.070977:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829199.070985:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829199.070991:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829199.071001:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4138 00000100:00080000:0.0:1787829199.071015:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829199.071025:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829199.071080:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564368000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829199.071109:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829199.071113:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829199.071115:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829199.071119:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4138 00000100:00080000:0.0:1787829199.071125:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829199.071132:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829199.071147:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829199.071157:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564368000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829199.071166:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564368128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829199.071177:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829199.071182:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829199.071184:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829199.071190:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4138 00000100:00080000:0.0:1787829199.071196:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829199.071202:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829199.071209:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829199.071222:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829199.071228:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564368256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829199.071241:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829199.071244:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829199.071246:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829199.071251:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4138 00000100:00080000:0.0:1787829199.071257:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829199.071263:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829199.071265:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829199/real 1787829199] req@ffff976882aec3c0 x1874670564368000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829199.071274:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829199.071293:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564368384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829199.071302:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829199.071303:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564368000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829199.071313:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829199.071319:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829199.071333:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4138 00000100:00080000:0.0:1787829199.071337:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829199.071338:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829199.071344:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829199.071352:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787829199.071358:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829199.071365:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564368512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829199.071368:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564368000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829199.071408:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564368128:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829199.071435:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829199.071447:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829199/real 1787829199] req@ffff976882aed680 x1874670564368128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829199.071459:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564368128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829199.071466:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829199.071469:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829199.071471:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564368128:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829199.071476:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564368256:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829199.071488:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829199.071494:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829199/real 1787829199] req@ffff976882aee580 x1874670564368256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829199.071502:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564368256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829199.071511:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829199.071514:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829199.071516:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564368256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829199.071519:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564368384:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829199.071530:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829199.071536:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829199/real 1787829199] req@ffff976882aeed00 x1874670564368384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829199.071544:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564368384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829199.071549:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829199.071551:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829199.071553:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564368384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829199.071557:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564368512:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829199.071568:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829199.071573:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829199/real 1787829199] req@ffff976882aef0c0 x1874670564368512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829199.071580:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564368512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829199.071586:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829199.071587:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829199.071590:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564368512:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829199.326977:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829199.326977:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829199.327060:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829200.094927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787829200.350989:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829200.350995:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829200.350996:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829201.119020:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829201.374992:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829201.375012:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829201.375109:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829202.142966:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829202.398975:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829202.398976:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829202.399012:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829203.166975:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829203.422961:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829203.422961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829203.423028:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829204.190952:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829204.190959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829204.190979:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829204.190986:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829204.191001:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4144 00000100:00080000:0.0:1787829204.191015:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829204.191026:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829204.191091:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564368640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829204.191128:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829204.191132:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829204.191135:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829204.191140:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4144 00000100:00080000:0.0:1787829204.191145:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829204.191150:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829204.191163:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829204.191176:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564368640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829204.191183:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564368768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829204.191195:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829204.191200:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829204.191202:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829204.191207:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4144 00000100:00080000:0.0:1787829204.191214:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829204.191218:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829204.191225:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829204.191238:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564368896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829204.191258:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829204.191261:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829204.191263:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829204.191267:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4144 00000100:00080000:0.0:1787829204.191271:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829204.191277:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829204.191278:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829204.191285:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829204.191299:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564369024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829204.191314:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829204.191318:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829204.191320:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829204.191324:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4144 00000100:00080000:0.0:1787829204.191329:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829204.191334:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829204/real 1787829204] req@ffff976882aee940 x1874670564368640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829204.191336:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829204.191344:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829204.191361:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564369152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829204.191365:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564368640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829204.191390:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829204.191394:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829204.191398:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564368640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829204.191421:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564368768:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829204.191441:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829204.191451:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829204/real 1787829204] req@ffff976882aef0c0 x1874670564368768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829204.191460:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564368768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829204.191466:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829204.191469:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829204.191471:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564368768:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829204.191475:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564368896:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829204.191487:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829204.191493:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829204/real 1787829204] req@ffff976882aeed00 x1874670564368896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829204.191506:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564368896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829204.191512:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829204.191514:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829204.191515:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564368896:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829204.191519:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564369024:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829204.191541:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829204.191548:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829204/real 1787829204] req@ffff976882aee580 x1874670564369024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829204.191556:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564369024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829204.191562:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829204.191564:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829204.191566:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564369024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829204.191570:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564369152:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829204.191582:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829204.191593:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829204/real 1787829204] req@ffff976882aed680 x1874670564369152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829204.191607:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564369152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829204.191623:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829204.191627:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829204.191632:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564369152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829204.446969:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829204.446970:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829204.447007:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829205.214967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829205.470917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829205.470932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829205.470967:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829206.238941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829206.494934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829206.494936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829206.494972:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829207.262952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829207.518998:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829207.518999:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829207.519070:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829208.286917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829208.542931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829208.542932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829208.542962:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829209.310938:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829209.310948:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829209.310951:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829209.310951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829209.310962:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4149 00000100:00080000:0.0:1787829209.310977:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829209.310984:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829209.311017:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564369280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829209.311039:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829209.311042:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829209.311044:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829209.311046:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4149 00000100:00080000:0.0:1787829209.311049:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829209.311051:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829209.311060:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829209.311070:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564369408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829209.311079:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829209.311079:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564369280:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829209.311085:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829209.311088:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829209.311092:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4149 00000100:00080000:0.0:1787829209.311095:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829209.311097:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829209.311100:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829209.311107:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564369536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829209.311113:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829209.311116:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829209.311117:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829209.311120:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4149 00000100:00080000:0.0:1787829209.311138:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829209.311140:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829209.311144:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1787829209.311150:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829209.311154:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564369664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829209.311161:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829209.311163:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829209.311165:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829209.311168:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4149 00000100:00080000:0.0:1787829209.311173:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829209.311175:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829209.311177:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829209.311184:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564369792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829209.311198:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829209/real 1787829209] req@ffff976882aec3c0 x1874670564369280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829209.311218:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564369280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829209.311229:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829209.311233:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829209.311237:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564369280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829209.311261:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564369408:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829209.311287:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829209.311303:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829209/real 1787829209] req@ffff976882aed680 x1874670564369408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829209.311317:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564369408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829209.311327:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829209.311331:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829209.311334:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564369408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829209.311351:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564369536:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829209.311374:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829209.311386:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829209/real 1787829209] req@ffff976882aee580 x1874670564369536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829209.311400:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564369536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829209.311409:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829209.311413:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829209.311416:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564369536:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829209.311423:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564369664:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829209.311444:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829209.311455:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829209/real 1787829209] req@ffff976882aeed00 x1874670564369664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829209.311467:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564369664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829209.311476:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829209.311480:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829209.311483:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564369664:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829209.311489:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564369792:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829209.311510:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829209.311523:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829209/real 1787829209] req@ffff976882aef0c0 x1874670564369792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829209.311535:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564369792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829209.311545:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829209.311549:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829209.311552:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564369792:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829209.566944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829209.566944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829209.567009:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829210.334939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829210.590927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829210.590931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829210.590942:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829211.358930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829211.614943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829211.614943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829211.615000:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829212.382921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829212.638900:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829212.638922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829212.638975:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829213.406972:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829213.662923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829213.662923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829213.662978:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829214.430932:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829214.430940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829214.430941:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829214.430947:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829214.430959:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4154 00000100:00080000:0.0:1787829214.430971:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829214.430978:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829214.431021:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564369920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829214.431045:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829214.431049:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829214.431050:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829214.431053:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4154 00000100:00080000:0.0:1787829214.431056:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829214.431059:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829214.431070:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829214.431078:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564369920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829214.431084:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564370048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829214.431094:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829214.431096:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829214.431097:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829214.431100:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4154 00000100:00080000:0.0:1787829214.431102:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829214.431104:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829214.431107:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829214.431115:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564370176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829214.431121:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829214.431123:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829214.431124:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829214.431127:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4154 00000100:00080000:0.0:1787829214.431145:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829214.431149:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829214.431153:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829214.431161:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564370304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829214.431167:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829214.431174:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829214.431177:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829214.431178:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829214.431182:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4154 00000100:00080000:0.0:1787829214.431184:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829214.431187:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829214.431200:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829214.431209:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564370432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829214.431217:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829214/real 1787829214] req@ffff976882aee940 x1874670564369920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829214.431233:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564369920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829214.431243:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829214.431246:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829214.431249:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564369920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829214.431271:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564370048:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829214.431288:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829214.431304:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829214/real 1787829214] req@ffff976882aef0c0 x1874670564370048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829214.431312:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564370048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829214.431319:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829214.431321:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829214.431323:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564370048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829214.431328:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564370176:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829214.431348:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829214.431360:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829214/real 1787829214] req@ffff976882aeed00 x1874670564370176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829214.431377:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564370176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829214.431385:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829214.431388:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829214.431391:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564370176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829214.431396:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564370304:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829214.431417:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829214.431429:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829214/real 1787829214] req@ffff976882aee580 x1874670564370304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829214.431442:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564370304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829214.431449:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829214.431452:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829214.431455:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564370304:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829214.431461:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564370432:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829214.431480:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829214.431490:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829214/real 1787829214] req@ffff976882aed680 x1874670564370432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829214.431502:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564370432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829214.431515:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829214.431519:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829214.431522:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564370432:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829214.686923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829214.686923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829214.686987:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829215.454973:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829215.710973:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829215.710992:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829215.711002:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829216.478934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829216.734908:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829216.734935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829216.734948:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829217.502925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829217.758899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829217.758899:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829217.758965:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829218.526895:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829218.782916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829218.782917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829218.783037:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829219.550922:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829219.550930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829219.550931:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829219.550937:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829219.550947:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4159 00000100:00080000:0.0:1787829219.550957:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829219.550963:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829219.550997:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564370560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829219.551019:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829219.551022:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829219.551024:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829219.551027:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4159 00000100:00080000:0.0:1787829219.551029:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829219.551032:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829219.551040:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829219.551052:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564370560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829219.551053:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564370688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829219.551066:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829219.551068:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829219.551069:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829219.551073:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4159 00000100:00080000:0.0:1787829219.551076:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829219.551080:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829219.551085:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829219.551099:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564370816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829219.551110:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829219.551113:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829219.551118:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829219.551122:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829219.551128:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4159 00000100:00080000:0.0:1787829219.551133:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829219.551137:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829219.551142:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829219.551150:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829219/real 1787829219] req@ffff976882aec3c0 x1874670564370560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829219.551155:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564370944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829219.551174:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829219.551176:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829219.551178:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829219.551183:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4159 00000100:00100000:1.0:1787829219.551184:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564370560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829219.551185:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829219.551190:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829219.551195:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829219.551205:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564371072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829219.551209:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829219.551217:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829219.551221:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564370560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829219.551242:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564370688:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829219.551262:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829219.551272:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829219/real 1787829219] req@ffff976882aed680 x1874670564370688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829219.551281:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564370688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829219.551287:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829219.551289:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829219.551291:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564370688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829219.551295:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564370816:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829219.551306:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829219.551312:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829219/real 1787829219] req@ffff976882aee580 x1874670564370816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829219.551319:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564370816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829219.551324:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829219.551326:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829219.551328:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564370816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829219.551331:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564370944:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829219.551341:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829219.551353:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829219/real 1787829219] req@ffff976882aeed00 x1874670564370944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829219.551360:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564370944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829219.551366:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829219.551368:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829219.551370:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564370944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829219.551374:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564371072:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829219.551384:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829219.551389:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829219/real 1787829219] req@ffff976882aef0c0 x1874670564371072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829219.551396:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564371072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829219.551401:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829219.551403:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829219.551405:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564371072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829219.806955:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829219.806955:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829219.807026:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829220.574911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829220.830911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829220.830931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829220.830954:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829221.598910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829221.854947:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829221.854947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829221.855007:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829222.622938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829222.878883:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829222.878887:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829222.878953:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829223.646944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829223.902892:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829223.902935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829223.902967:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829224.670927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829224.670946:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829224.670957:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829224.670962:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829224.670972:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4164 00000100:00080000:2.0:1787829224.670982:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829224.670990:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829224.671026:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564371200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829224.671057:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829224.671061:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829224.671063:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829224.671068:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4164 00000100:00080000:2.0:1787829224.671073:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829224.671078:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829224.671088:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829224.671098:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564371200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829224.671106:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564371328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829224.671122:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829224.671126:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829224.671129:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829224.671135:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4164 00000100:00080000:2.0:1787829224.671139:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829224.671146:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829224.671152:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829224.671166:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564371456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829224.671177:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829224.671180:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829224.671183:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829224.671190:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4164 00000800:00000100:1.0:1787829224.671193:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829224.671204:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829224.671212:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829224.671219:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829224.671234:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564371584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829224.671244:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829224/real 1787829224] req@ffff976889cf8f00 x1874670564371200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829224.671248:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829224.671267:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829224.671271:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829224.671278:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564371200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829224.671284:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4164 00000100:00080000:2.0:1787829224.671298:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829224.671303:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829224.671307:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:1.0:1787829224.671311:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829224.671321:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564371200:192.168.203.139@tcp:250:kworker.0 00000008:00100000:2.0:1787829224.671322:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829224.671347:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564371712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829224.671364:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564371328:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829224.671391:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829224.671405:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829224/real 1787829224] req@ffff976889cf8000 x1874670564371328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829224.671420:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564371328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829224.671430:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829224.671433:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829224.671437:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564371328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829224.671452:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564371456:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829224.671474:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829224.671487:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829224/real 1787829224] req@ffff976889cfb0c0 x1874670564371456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829224.671507:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564371456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829224.671518:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829224.671521:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829224.671525:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564371456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829224.671532:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564371584:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829224.671555:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829224.671569:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829224/real 1787829224] req@ffff976889cf9e00 x1874670564371584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829224.671582:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564371584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829224.671592:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829224.671596:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829224.671600:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564371584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829224.671608:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564371712:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829224.671629:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829224.671642:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829224/real 1787829224] req@ffff976889cfa580 x1874670564371712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829224.671655:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564371712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829224.671672:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829224.671676:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829224.671679:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564371712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829224.926889:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829224.926933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829224.926963:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829225.694928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829225.950926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829225.950929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829225.950952:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829226.718943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829226.974875:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829226.974893:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829226.974957:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829227.742931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829227.998877:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829227.998893:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829227.998918:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829228.766950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829229.022945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829229.022945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829229.023022:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829229.790923:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829229.790926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829229.790936:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829229.790942:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829229.790954:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4169 00000100:00080000:2.0:1787829229.790968:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829229.790977:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829229.791027:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564371840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829229.791057:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829229.791060:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829229.791063:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829229.791068:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4169 00000100:00080000:2.0:1787829229.791072:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829229.791078:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829229.791089:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829229.791095:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564371840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829229.791108:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564371968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829229.791121:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829229.791125:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829229.791127:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829229.791131:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4169 00000100:00080000:2.0:1787829229.791135:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829229.791140:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829229.791145:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829229.791156:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564372096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829229.791167:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829229.791168:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829229.791173:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829229.791175:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829229.791181:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4169 00000100:00080000:2.0:1787829229.791186:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829229.791191:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829229.791196:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829229.791208:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564372224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829229.791218:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829229.791223:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829229/real 1787829229] req@ffff976889cfbc00 x1874670564371840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829229.791224:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829229.791227:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829229.791236:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4169 00000100:00080000:2.0:1787829229.791242:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829229.791247:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829229.791250:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564371840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829229.791255:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829229.791262:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829229.791267:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829229.791272:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564371840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829229.791272:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564372352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829229.791302:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564371968:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829229.791321:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829229.791331:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829229/real 1787829229] req@ffff976889cfa580 x1874670564371968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829229.791339:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564371968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829229.791353:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829229.791358:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829229.791361:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564371968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829229.791365:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564372096:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829229.791380:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829229.791389:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829229/real 1787829229] req@ffff976889cf9e00 x1874670564372096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829229.791399:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564372096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829229.791405:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829229.791408:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829229.791410:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564372096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829229.791413:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564372224:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829229.791424:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829229.791430:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829229/real 1787829229] req@ffff976889cfb0c0 x1874670564372224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829229.791437:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564372224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829229.791442:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829229.791444:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829229.791446:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564372224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829229.791450:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564372352:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829229.791462:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829229.791468:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829229/real 1787829229] req@ffff976889cf8000 x1874670564372352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829229.791475:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564372352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829229.791479:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829229.791481:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829229.791483:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564372352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829230.046919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829230.046919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829230.046946:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829230.814939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829231.070944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829231.070980:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829231.070987:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829231.838936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829232.094913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829232.094913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829232.094938:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829232.862998:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829233.118947:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829233.118947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829233.119011:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829233.886935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829234.142922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829234.142931:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829234.142932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829234.910954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829234.911119:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829234.911125:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829234.911130:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829234.911139:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4174 00000100:00080000:0.0:1787829234.911152:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829234.911159:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829234.911213:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564372480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829234.911238:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829234.911241:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829234.911242:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829234.911246:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4174 00000100:00080000:0.0:1787829234.911248:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829234.911251:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829234.911261:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829234.911272:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564372608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829234.911279:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829234.911287:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829234.911289:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829234.911293:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4174 00000100:00080000:0.0:1787829234.911297:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829234.911299:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829234.911303:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829234.911310:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564372736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829234.911319:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829234.911322:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829234.911323:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829234.911328:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4174 00000100:00080000:0.0:1787829234.911332:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829234.911338:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564372480:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829234.911339:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829234.911352:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829234.911361:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564372864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829234.911369:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829234.911372:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829234.911374:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829234.911377:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4174 00000100:00080000:0.0:1787829234.911379:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829234.911383:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829234.911386:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829234.911397:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564372992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829234.911417:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829234.911467:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829234/real 1787829234] req@ffff976882aee940 x1874670564372480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829234.911492:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564372480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829234.911504:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829234.911509:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829234.911513:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564372480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829234.911523:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564372608:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829234.911549:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829234.911563:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829234/real 1787829234] req@ffff976882aef0c0 x1874670564372608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829234.911576:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564372608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829234.911586:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829234.911589:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829234.911593:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564372608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829234.911619:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564372736:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829234.911637:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829234.911645:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829234/real 1787829234] req@ffff976882aeed00 x1874670564372736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829234.911653:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564372736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829234.911685:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829234.911690:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829234.911693:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564372736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829234.911700:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564372864:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829234.911716:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829234.911724:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829234/real 1787829234] req@ffff976882aee580 x1874670564372864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829234.911731:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564372864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829234.911737:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829234.911739:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829234.911741:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564372864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829234.911745:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564372992:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829234.911755:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829234.911761:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829234/real 1787829234] req@ffff976882aed680 x1874670564372992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829234.911768:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564372992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829234.911772:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829234.911774:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829234.911777:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564372992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829235.166981:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829235.167067:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829235.167073:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829235.934942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829236.190943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829236.191001:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829236.191025:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829236.958933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829237.214974:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829237.215024:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829237.215086:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829237.982906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829238.238935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829238.238942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829238.238945:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829239.006941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829239.262960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829239.262966:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829239.263040:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829240.030895:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829240.030906:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829240.030914:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829240.030918:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829240.030927:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4179 00000100:00080000:0.0:1787829240.030936:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829240.030941:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829240.030977:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564373120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829240.031008:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829240.031012:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829240.031015:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829240.031020:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4179 00000100:00080000:0.0:1787829240.031025:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829240.031029:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829240.031038:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829240.031043:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564373120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829240.031055:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564373248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829240.031066:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829240.031070:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829240.031072:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829240.031077:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4179 00000100:00080000:0.0:1787829240.031082:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829240.031087:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829240.031093:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829240.031108:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564373376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829240.031121:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829240.031124:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829240.031126:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829240.031131:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4179 00000100:00080000:0.0:1787829240.031134:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829240.031138:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829240.031143:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564373248:192.168.203.139@tcp:38:kworker.0 00000008:00100000:0.0:1787829240.031144:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829240.031158:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564373504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829240.031166:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829240.031169:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829240.031170:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829240.031174:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4179 00000100:00080000:0.0:1787829240.031177:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829240.031179:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829240.031183:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829240.031186:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564373376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829240.031195:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564373632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829240.031218:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564373504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829240.031235:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564373632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829240.286909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829240.286909:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829240.286935:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829241.055043:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829241.310918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829241.310918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829241.310961:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829242.078936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829242.334939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829242.334941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829242.334977:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829243.102902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:1.0:1787829243.103049:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:1.0:1787829243.103085:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787829243.103099:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787829243.103137:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787829243.103147:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787829243.103158:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787829243.103166:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:2.0:1787829243.103195:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829240/real 1787829243] req@ffff976882aeed00 x1874670564373504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829243.103222:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564373504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829243.103234:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829243.103239:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829243.103243:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564373504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829243.103251:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829240/real 1787829243] req@ffff976882aef0c0 x1874670564373632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829243.103278:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564373632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829243.103286:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829243.103289:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829243.103292:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564373632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829243.103299:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829240/real 1787829243] req@ffff976882aed680 x1874670564373248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829243.103310:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564373248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829243.103319:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829243.103322:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829243.103325:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564373248:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829243.103331:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829240/real 1787829243] req@ffff976882aee580 x1874670564373376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829243.103342:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564373376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829243.103350:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829243.103353:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829243.103355:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564373376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829243.103361:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829240/real 1787829243] req@ffff976882aec3c0 x1874670564373120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829243.103373:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564373120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829243.103380:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829243.103384:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829243.103387:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564373120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829243.358965:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829243.358977:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829243.359026:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829244.126919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829244.382935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829244.383045:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829244.383047:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00080000:0.0:1787829245.150946:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829245.150946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829245.150957:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829245.150962:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829245.150973:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4185 00000100:00080000:0.0:1787829245.150984:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829245.150991:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829245.151041:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564373760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829245.151084:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829245.151088:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829245.151090:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829245.151094:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4185 00000100:00080000:0.0:1787829245.151098:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829245.151102:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829245.151113:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829245.151129:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564373888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829245.151133:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564373760:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829245.151149:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829245.151153:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829245.151155:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829245.151160:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4185 00000100:00080000:0.0:1787829245.151165:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829245.151170:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829245.151175:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829245.151188:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564374016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829245.151199:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829245.151203:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829245.151206:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829245.151208:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829245.151215:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4185 00000100:00080000:0.0:1787829245.151221:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829245.151226:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829245.151232:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829245.151246:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564374144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829245.151257:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829245/real 1787829245] req@ffff976882aee940 x1874670564373760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829245.151259:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829245.151282:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829245.151284:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564373760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829245.151286:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829245.151296:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4185 00000100:00080000:0.0:1787829245.151303:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829245.151306:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829245.151311:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:2.0:1787829245.151316:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000008:00100000:0.0:1787829245.151321:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829245.151323:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564373760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829245.151338:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564374272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829245.151354:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564373888:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829245.151384:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829245.151413:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829245/real 1787829245] req@ffff976882aec3c0 x1874670564373888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829245.151426:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564373888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829245.151435:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829245.151439:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829245.151442:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564373888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829245.151448:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564374016:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829245.151471:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829245.151484:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829245/real 1787829245] req@ffff976882aee580 x1874670564374016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829245.151497:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564374016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829245.151506:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829245.151510:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829245.151514:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564374016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829245.151520:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564374144:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829245.151542:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829245.151554:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829245/real 1787829245] req@ffff976882aed680 x1874670564374144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829245.151567:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564374144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829245.151576:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829245.151580:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829245.151583:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564374144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829245.151615:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564374272:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829245.151636:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829245.151646:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829245/real 1787829245] req@ffff976882aef0c0 x1874670564374272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829245.151659:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564374272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829245.151667:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829245.151671:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829245.151674:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564374272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829245.406907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829245.406907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829245.406961:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829246.174938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829246.430967:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829246.430969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829246.430998:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829247.198917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829247.455002:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829247.455038:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829247.455112:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829248.222943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829248.478999:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829248.479001:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829248.479031:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829249.246912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829249.502925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829249.502962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829249.503189:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829250.270953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829250.270981:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829250.270987:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829250.270990:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829250.271000:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4190 00000100:00080000:0.0:1787829250.271010:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829250.271018:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829250.271059:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564374400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829250.271085:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829250.271089:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829250.271091:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829250.271095:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4190 00000100:00080000:0.0:1787829250.271098:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829250.271103:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829250.271113:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829250.271121:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564374400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829250.271131:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564374528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829250.271144:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829250.271148:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829250.271150:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829250.271154:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4190 00000100:00080000:0.0:1787829250.271158:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829250.271162:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829250.271167:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829250.271181:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564374656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829250.271182:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829250.271193:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829250.271197:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829250.271199:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829250.271205:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4190 00000100:00080000:0.0:1787829250.271209:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829250.271213:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829250.271219:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829250.271232:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564374784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829250.271237:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829250/real 1787829250] req@ffff976882aeed00 x1874670564374400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829250.271253:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829250.271258:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829250.271261:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564374400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829250.271267:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829250.271271:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829250.271275:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4190 00000100:00080000:2.0:1787829250.271277:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:0.0:1787829250.271284:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829250.271284:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564374400:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829250.271293:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829250.271298:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829250.271309:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564374528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829250.271311:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564374912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829250.271341:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829250.271354:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829250/real 1787829250] req@ffff976882aef0c0 x1874670564374528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829250.271363:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564374528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829250.271369:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829250.271371:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829250.271374:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564374528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829250.271377:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564374656:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829250.271390:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829250.271396:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829250/real 1787829250] req@ffff976882aed680 x1874670564374656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829250.271404:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564374656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829250.271409:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829250.271411:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829250.271413:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564374656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829250.271417:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564374784:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829250.271427:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829250.271433:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829250/real 1787829250] req@ffff976882aee580 x1874670564374784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829250.271445:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564374784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829250.271451:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829250.271452:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829250.271454:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564374784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829250.271464:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564374912:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829250.271474:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829250.271480:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829250/real 1787829250] req@ffff976882aec3c0 x1874670564374912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829250.271487:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564374912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829250.271492:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829250.271494:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829250.271496:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564374912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829250.526945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829250.526945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829250.526971:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829251.294923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829251.550920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829251.550928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829251.550932:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829252.318955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829252.574956:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829252.574998:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829252.575008:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787829253.342957:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829253.598906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829253.598945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829253.598990:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829254.366935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829254.622970:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829254.623003:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829254.623009:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829255.390932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829255.390939:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829255.390972:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829255.390976:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829255.390988:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4195 00000100:00080000:0.0:1787829255.390997:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829255.391003:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829255.391037:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564375040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829255.391060:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829255.391063:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829255.391064:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829255.391067:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4195 00000100:00080000:0.0:1787829255.391069:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829255.391072:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829255.391081:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829255.391091:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564375168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829255.391108:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564375040:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829255.391114:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829255.391118:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829255.391120:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829255.391123:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4195 00000100:00080000:0.0:1787829255.391125:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829255.391127:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829255.391131:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829255.391139:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564375296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829255.391145:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829255.391147:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829255.391148:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829255.391151:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4195 00000100:00080000:0.0:1787829255.391154:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829255.391157:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829255.391161:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829255.391169:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564375424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829255.391177:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829255.391178:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829255.391183:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829255.391184:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829255.391188:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4195 00000100:00080000:0.0:1787829255.391191:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829255.391194:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829255.391197:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829255.391205:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564375552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829255.391219:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829255/real 1787829255] req@ffff976882aee940 x1874670564375040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829255.391234:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564375040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829255.391242:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829255.391245:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829255.391248:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564375040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829255.391265:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564375168:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829255.391280:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829255.391289:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829255/real 1787829255] req@ffff976882aec3c0 x1874670564375168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829255.391297:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564375168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829255.391302:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829255.391305:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829255.391307:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564375168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829255.391310:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564375296:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829255.391320:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829255.391326:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829255/real 1787829255] req@ffff976882aee580 x1874670564375296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829255.391333:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564375296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829255.391343:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829255.391345:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829255.391347:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564375296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829255.391350:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564375424:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829255.391361:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829255.391366:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829255/real 1787829255] req@ffff976882aed680 x1874670564375424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829255.391373:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564375424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829255.391387:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829255.391389:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829255.391391:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564375424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829255.391394:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564375552:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829255.391404:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829255.391409:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829255/real 1787829255] req@ffff976882aef0c0 x1874670564375552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829255.391416:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564375552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829255.391421:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829255.391422:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829255.391425:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564375552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829255.646932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829255.646938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829255.647064:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829256.414928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829256.670901:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829256.670922:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829256.670934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829257.438888:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829257.694889:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829257.694914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829257.695012:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829258.462922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829258.718899:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829258.718950:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829258.718978:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829259.486977:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829259.742969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829259.743007:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829259.743212:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829260.510959:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829260.510959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829260.510974:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829260.510983:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829260.511014:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4200 00000100:00080000:0.0:1787829260.511033:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829260.511045:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829260.511121:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564375680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829260.511155:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829260.511159:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829260.511161:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829260.511165:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4200 00000100:00080000:0.0:1787829260.511169:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829260.511174:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829260.511184:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829260.511201:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564375808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829260.511212:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829260.511216:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787829260.511217:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564375680:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829260.511219:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829260.511228:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4200 00000100:00080000:0.0:1787829260.511231:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829260.511234:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829260.511239:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829260.511251:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564375936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829260.511258:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829260.511261:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829260.511264:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829260.511268:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4200 00000100:00080000:0.0:1787829260.511272:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829260.511275:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829260.511280:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829260.511289:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564376064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829260.511297:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829260.511300:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829260.511301:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829260.511304:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4200 00000100:00080000:0.0:1787829260.511308:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829260.511310:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829260.511315:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829260.511321:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829260.511331:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564376192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829260.511368:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829260/real 1787829260] req@ffff976882aeed00 x1874670564375680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829260.511387:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564375680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829260.511398:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829260.511402:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829260.511406:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564375680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829260.511434:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564375808:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829260.511453:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829260.511465:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829260/real 1787829260] req@ffff976882aef0c0 x1874670564375808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829260.511475:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564375808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829260.511500:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829260.511504:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829260.511508:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564375808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829260.511514:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564375936:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829260.511530:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829260.511539:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829260/real 1787829260] req@ffff976882aed680 x1874670564375936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829260.511549:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564375936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829260.511556:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829260.511559:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829260.511562:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564375936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829260.511566:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564376064:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829260.511580:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829260.511587:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829260/real 1787829260] req@ffff976882aee580 x1874670564376064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829260.511596:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564376064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829260.511603:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829260.511606:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829260.511608:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564376064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829260.511612:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564376192:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829260.511625:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829260.511633:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829260/real 1787829260] req@ffff976882aec3c0 x1874670564376192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829260.511642:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564376192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829260.511649:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829260.511651:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829260.511653:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564376192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829260.766953:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829260.766953:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829260.767058:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829261.534971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829261.790938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829261.790996:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829261.791026:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829262.558892:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829262.814899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829262.814904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829262.814952:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829263.582998:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829263.838928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829263.838963:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829263.839021:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829264.606907:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829264.862908:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829264.862912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829264.862967:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829265.630989:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829265.630994:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829265.631008:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829265.631016:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829265.631046:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4205 00000100:00080000:0.0:1787829265.631066:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829265.631078:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829265.631130:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564376320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829265.631160:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829265.631164:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829265.631166:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829265.631170:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4205 00000100:00080000:0.0:1787829265.631174:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829265.631178:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829265.631190:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829265.631208:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564376448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829265.631210:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564376320:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829265.631231:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829265.631234:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829265.631235:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829265.631238:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4205 00000100:00080000:0.0:1787829265.631240:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829265.631243:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829265.631247:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829265.631257:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564376576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829265.631265:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829265.631269:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829265.631271:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829265.631275:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4205 00000100:00080000:0.0:1787829265.631280:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829265.631283:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829265.631286:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829265.631296:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564376704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829265.631304:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829265.631311:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1787829265.631312:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829265.631315:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829265.631322:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4205 00000100:00080000:0.0:1787829265.631326:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829265.631330:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829265.631334:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829265.631344:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564376832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829265.631383:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829265/real 1787829265] req@ffff976882aee940 x1874670564376320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829265.631402:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564376320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829265.631413:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829265.631417:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829265.631421:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564376320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829265.631465:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564376448:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829265.631495:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829265.631508:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829265/real 1787829265] req@ffff976882aec3c0 x1874670564376448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829265.631520:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564376448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829265.631528:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829265.631531:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829265.631534:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564376448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829265.631539:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564376576:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829265.631554:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829265.631562:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829265/real 1787829265] req@ffff976882aee580 x1874670564376576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829265.631572:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564376576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829265.631579:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829265.631581:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829265.631584:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564376576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829265.631588:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564376704:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829265.631603:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829265.631610:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829265/real 1787829265] req@ffff976882aed680 x1874670564376704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829265.631627:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564376704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829265.631634:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829265.631636:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829265.631639:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564376704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829265.631643:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564376832:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829265.631657:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829265.631664:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829265/real 1787829265] req@ffff976882aef0c0 x1874670564376832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829265.631674:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564376832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829265.631681:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829265.631683:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829265.631686:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564376832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829265.886945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829265.886987:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829265.887113:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829266.654967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829266.910904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829266.910939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829266.910948:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829267.678992:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829267.934975:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829267.935008:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829267.935163:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829268.702988:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829268.958907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829268.958941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829268.959053:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829269.726955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829269.982968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829269.982984:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829269.983046:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829270.750926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829270.750946:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829270.750954:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829270.750959:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829270.750968:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4210 00000100:00080000:0.0:1787829270.750978:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829270.750987:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829270.751022:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564376960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829270.751046:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829270.751049:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829270.751050:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829270.751053:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4210 00000100:00080000:0.0:1787829270.751055:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829270.751057:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829270.751065:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829270.751076:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564377088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829270.751089:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829270.751097:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787829270.751098:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564376960:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829270.751100:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829270.751111:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4210 00000100:00080000:0.0:1787829270.751114:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829270.751116:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829270.751120:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829270.751131:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564377216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829270.751142:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829270.751145:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829270.751146:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829270.751149:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4210 00000100:00080000:0.0:1787829270.751151:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829270.751154:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829270.751159:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829270.751168:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564377344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829270.751175:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829270.751179:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829270.751182:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829270.751186:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4210 00000800:00000100:2.0:1787829270.751191:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829270.751201:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829270.751209:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829270.751216:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829270.751233:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564377472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829270.751244:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829270/real 1787829270] req@ffff976882aeed00 x1874670564376960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829270.751260:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564376960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829270.751272:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829270.751277:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829270.751281:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564376960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829270.751305:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564377088:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829270.751333:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829270.751342:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829270/real 1787829270] req@ffff976882aef0c0 x1874670564377088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829270.751350:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564377088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829270.751356:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829270.751358:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829270.751361:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564377088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829270.751364:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564377216:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829270.751376:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829270.751382:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829270/real 1787829270] req@ffff976882aed680 x1874670564377216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829270.751400:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564377216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829270.751425:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829270.751428:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829270.751431:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564377216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829270.751435:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564377344:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829270.751452:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829270.751459:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829270/real 1787829270] req@ffff976882aee580 x1874670564377344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829270.751467:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564377344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829270.751473:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829270.751475:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829270.751477:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564377344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829270.751481:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564377472:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829270.751492:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829270.751497:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829270/real 1787829270] req@ffff976882aec3c0 x1874670564377472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829270.751504:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564377472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829270.751509:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829270.751511:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829270.751514:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564377472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829271.006905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829271.006922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829271.006984:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829271.774938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829272.030939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829272.030942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829272.030997:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829272.798986:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829273.054916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829273.054938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829273.054984:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829273.822929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829274.078897:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829274.078949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829274.079023:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829274.846928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829275.102962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829275.102962:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829275.103044:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829275.870958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829275.871013:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829275.871021:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829275.871026:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829275.871037:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4215 00000100:00080000:2.0:1787829275.871051:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829275.871061:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829275.871110:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564377600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829275.871158:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829275.871162:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829275.871165:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829275.871170:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4215 00000100:00080000:2.0:1787829275.871175:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829275.871182:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829275.871197:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829275.871212:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564377600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829275.871219:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564377728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829275.871269:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829275.871276:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829275.871279:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829275.871287:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4215 00000100:00080000:2.0:1787829275.871293:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829275.871298:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829275.871311:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829275.871345:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564377856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829275.871357:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829275.871362:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829275.871390:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829275.871391:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829275.871403:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4215 00000100:00080000:2.0:1787829275.871410:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829275.871440:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829275.871445:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829275.871456:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564377984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829275.871463:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829275.871466:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829275.871467:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829275.871471:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4215 00000100:00080000:2.0:1787829275.871474:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829275.871478:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829275/real 1787829275] req@ffff976889cf8f00 x1874670564377600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829275.871479:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829275.871486:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829275.871501:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564378112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829275.871506:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564377600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829275.871523:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829275.871526:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829275.871530:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564377600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829275.871567:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564377728:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829275.871587:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829275.871598:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829275/real 1787829275] req@ffff976889cfa940 x1874670564377728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829275.871616:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564377728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829275.871623:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829275.871625:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829275.871627:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564377728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829275.871631:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564377856:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829275.871644:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829275.871656:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829275/real 1787829275] req@ffff976889cfb480 x1874670564377856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829275.871685:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564377856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829275.871695:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829275.871699:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829275.871703:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564377856:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829275.871710:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564377984:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829275.871731:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829275.871743:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829275/real 1787829275] req@ffff976889cfad00 x1874670564377984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829275.871764:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564377984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829275.871773:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829275.871776:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829275.871779:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564377984:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829275.871784:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564378112:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829275.871798:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829275.871812:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829275/real 1787829275] req@ffff976889cfa1c0 x1874670564378112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829275.871869:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564378112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829275.871891:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829275.871896:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829275.871904:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564378112:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829276.126954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829276.126992:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829276.127002:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829276.894917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829277.151020:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829277.151085:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829277.151117:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829277.918935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829278.174950:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829278.174952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829278.174977:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829278.942935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829279.198952:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829279.198952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829279.199094:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829279.966938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829280.222946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829280.222946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829280.223000:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829280.990940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829280.990940:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829280.990953:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829280.990959:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829280.990969:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4220 00000100:00080000:2.0:1787829280.990981:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829280.990988:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829280.991033:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564378240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829280.991055:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829280.991057:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829280.991059:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829280.991062:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4220 00000100:00080000:2.0:1787829280.991065:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829280.991069:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829280.991079:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829280.991089:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564378368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829280.991097:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564378240:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829280.991100:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829280.991107:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829280.991108:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829280.991112:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4220 00000100:00080000:2.0:1787829280.991114:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829280.991116:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829280.991119:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829280.991127:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564378496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829280.991133:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829280.991135:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829280.991136:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829280.991140:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4220 00000100:00080000:2.0:1787829280.991142:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829280.991145:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829280.991148:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829280.991156:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564378624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829280.991162:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829280.991164:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829280.991165:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829280.991167:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4220 00000100:00080000:2.0:1787829280.991170:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829280.991173:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829280.991177:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829280.991185:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564378752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829280.991191:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829280.991248:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829280/real 1787829280] req@ffff976889cf92c0 x1874670564378240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829280.991267:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564378240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829280.991280:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829280.991284:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829280.991288:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564378240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829280.991318:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564378368:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829280.991358:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829280.991374:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829280/real 1787829280] req@ffff976889cfa1c0 x1874670564378368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829280.991390:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564378368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829280.991397:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829280.991399:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829280.991402:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564378368:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829280.991407:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564378496:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829280.991422:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829280.991429:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829280/real 1787829280] req@ffff976889cfad00 x1874670564378496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829280.991445:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564378496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829280.991451:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829280.991454:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829280.991456:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564378496:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829280.991460:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564378624:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829280.991472:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829280.991479:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829280/real 1787829280] req@ffff976889cfb480 x1874670564378624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829280.991486:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564378624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829280.991496:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829280.991498:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829280.991501:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564378624:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829280.991504:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564378752:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829280.991514:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829280.991519:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829280/real 1787829280] req@ffff976889cfa940 x1874670564378752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829280.991526:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564378752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829280.991532:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829280.991534:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829280.991535:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564378752:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829281.246895:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829281.246911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829281.247029:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829282.014981:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829282.270965:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829282.270965:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829282.270997:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829283.038911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829283.294913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829283.294922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829283.295000:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829284.062949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829284.318919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829284.318920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829284.318959:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829285.086970:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829285.342944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829285.342946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829285.343027:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829286.110929:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829286.110941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829286.110941:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829286.110948:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829286.110960:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4226 00000100:00080000:3.0:1787829286.110972:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829286.110986:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829286.111032:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564378880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829286.111057:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829286.111060:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829286.111062:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829286.111065:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4226 00000100:00080000:3.0:1787829286.111067:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829286.111070:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829286.111080:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829286.111092:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564379008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829286.111116:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829286.111118:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564378880:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787829286.111120:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829286.111125:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829286.111131:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4226 00000100:00080000:3.0:1787829286.111134:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829286.111136:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829286.111140:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829286.111150:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564379136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829286.111156:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829286.111159:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829286.111161:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829286.111165:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4226 00000100:00080000:3.0:1787829286.111170:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829286.111174:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829286.111178:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829286.111186:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564379264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829286.111194:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829286.111198:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829286.111201:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829286.111206:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787829286.111212:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4226 00000100:00080000:3.0:1787829286.111217:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829286.111221:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829286.111226:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829286.111237:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564379392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829286.111265:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829286/real 1787829286] req@ffff97688cb5d2c0 x1874670564378880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829286.111289:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564378880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829286.111336:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829286.111341:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829286.111344:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564378880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829286.111374:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564379008:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829286.111399:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829286.111412:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829286/real 1787829286] req@ffff97688cb5f0c0 x1874670564379008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829286.111423:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564379008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829286.111430:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829286.111433:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829286.111436:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564379008:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829286.111441:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564379136:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829286.111455:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829286.111463:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829286/real 1787829286] req@ffff97688cb5c3c0 x1874670564379136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829286.111472:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564379136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829286.111485:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829286.111488:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829286.111490:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564379136:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829286.111495:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564379264:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829286.111508:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829286.111516:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829286/real 1787829286] req@ffff97688cb5c780 x1874670564379264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829286.111524:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564379264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829286.111531:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829286.111534:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829286.111536:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564379264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829286.111541:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564379392:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829286.111555:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829286.111567:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829286/real 1787829286] req@ffff97688cb5e1c0 x1874670564379392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829286.111583:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564379392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829286.111593:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829286.111598:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829286.111603:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564379392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829286.366903:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829286.366903:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829286.366977:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829287.134950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829287.390972:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829287.390976:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829287.391008:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829288.158950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829288.414899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829288.414957:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829288.414972:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829289.182948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829289.438890:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829289.438905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829289.438934:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829290.206940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829290.462893:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829290.462907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829290.462928:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829291.230968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829291.230978:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829291.230984:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829291.230990:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829291.231001:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4231 00000100:00080000:2.0:1787829291.231014:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829291.231023:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829291.231073:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564379520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829291.231099:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829291.231103:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829291.231105:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829291.231110:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4231 00000100:00080000:2.0:1787829291.231114:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829291.231121:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829291.231130:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564379520:192.168.203.139@tcp:250:kworker.0 00000008:00100000:2.0:1787829291.231133:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829291.231147:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564379648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829291.231157:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829291.231160:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829291.231162:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829291.231166:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4231 00000100:00080000:2.0:1787829291.231169:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829291.231172:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829291.231177:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829291.231186:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564379776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829291.231195:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829291.231198:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829291.231200:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829291.231204:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829291.231204:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4231 00000100:00080000:2.0:1787829291.231207:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829291.231210:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829291.231215:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829291.231224:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564379904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829291.231232:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829291.231235:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829291.231237:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829291.231240:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4231 00000100:00080000:2.0:1787829291.231243:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829291.231247:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829291.231250:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829291/real 1787829291] req@ffff976889cf8f00 x1874670564379520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829291.231251:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829291.231261:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564380032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829291.231282:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564379520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829291.231293:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829291.231298:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829291.231302:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564379520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829291.231343:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564379648:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829291.231364:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829291.231374:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829291/real 1787829291] req@ffff976889cf8000 x1874670564379648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829291.231383:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564379648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829291.231389:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829291.231393:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829291.231395:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564379648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829291.231399:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564379776:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829291.231411:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829291.231417:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829291/real 1787829291] req@ffff976889cfb0c0 x1874670564379776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829291.231425:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564379776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829291.231430:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829291.231432:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829291.231434:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564379776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829291.231437:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564379904:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829291.231448:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829291.231454:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829291/real 1787829291] req@ffff976889cf9e00 x1874670564379904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829291.231461:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564379904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829291.231467:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829291.231469:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829291.231471:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564379904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829291.231474:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564380032:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829291.231486:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829291.231491:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829291/real 1787829291] req@ffff976889cfa580 x1874670564380032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829291.231499:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564380032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829291.231504:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829291.231506:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829291.231508:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564380032:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829291.486932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829291.486935:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829291.486945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829292.254931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829292.510903:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829292.510909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829292.510928:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829293.278927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829293.534893:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829293.534940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829293.534971:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829294.303023:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829294.558894:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829294.558906:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829294.558917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829295.326918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829295.582948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829295.582948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829295.583018:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829296.350912:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829296.350916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829296.350927:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829296.350933:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829296.350944:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4236 00000100:00080000:2.0:1787829296.350959:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829296.350969:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829296.351021:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564380160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829296.351051:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829296.351055:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829296.351058:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829296.351063:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4236 00000100:00080000:2.0:1787829296.351069:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829296.351075:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829296.351083:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564380160:192.168.203.139@tcp:250:kworker.0 00000008:00100000:2.0:1787829296.351087:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829296.351109:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564380288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829296.351122:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829296.351127:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829296.351129:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829296.351134:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4236 00000100:00080000:2.0:1787829296.351137:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829296.351140:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829296.351145:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829296.351157:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564380416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829296.351160:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829296.351169:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829296.351172:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829296.351173:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829296.351177:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4236 00000100:00080000:2.0:1787829296.351179:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829296.351181:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829296.351185:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829296.351192:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564380544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829296.351204:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829296.351207:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829296.351208:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829296.351210:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4236 00000100:00080000:2.0:1787829296.351213:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829296.351215:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829296.351218:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829296.351240:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564380672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829296.351247:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829296/real 1787829296] req@ffff976889cfbc00 x1874670564380160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829296.351271:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564380160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829296.351284:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829296.351289:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829296.351294:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564380160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829296.351323:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564380288:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829296.351351:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829296.351378:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829296/real 1787829296] req@ffff976889cfa580 x1874670564380288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829296.351393:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564380288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829296.351403:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829296.351407:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829296.351411:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564380288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829296.351418:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564380416:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829296.351440:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829296.351454:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829296/real 1787829296] req@ffff976889cf9e00 x1874670564380416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829296.351466:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564380416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829296.351480:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829296.351484:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829296.351488:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564380416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829296.351495:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564380544:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829296.351518:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829296.351532:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829296/real 1787829296] req@ffff976889cfb0c0 x1874670564380544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829296.351545:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564380544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829296.351555:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829296.351558:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829296.351562:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564380544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829296.351567:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564380672:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829296.351583:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829296.351591:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829296/real 1787829296] req@ffff976889cf8000 x1874670564380672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829296.351599:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564380672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829296.351605:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829296.351607:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829296.351609:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564380672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829296.606917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829296.606921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829296.606945:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829297.374978:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829297.630952:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829297.630952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829297.630987:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829298.398957:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829298.654938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829298.654939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829298.654945:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829299.422905:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829299.678956:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829299.678964:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829299.679045:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829300.446942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829300.702912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829300.702920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829300.702973:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829301.470925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829301.470965:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829301.470977:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829301.470981:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829301.470992:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4241 00000100:00080000:3.0:1787829301.471005:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829301.471013:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829301.471059:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564380800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829301.471087:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829301.471090:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829301.471092:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829301.471096:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4241 00000100:00080000:3.0:1787829301.471100:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829301.471104:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829301.471114:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829301.471131:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564380800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829301.471132:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564380928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829301.471160:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829301.471164:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829301.471165:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829301.471171:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4241 00000100:00080000:3.0:1787829301.471174:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829301.471177:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829301.471181:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829301.471207:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564381056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829301.471216:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829301.471220:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829301.471222:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829301.471226:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4241 00000100:00080000:3.0:1787829301.471230:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829301.471233:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829301.471238:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829301.471249:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564381184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829301.471259:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787829301.471275:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829301.471278:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829301.471280:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829301.471287:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4241 00000100:00080000:3.0:1787829301.471291:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829301.471293:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829301.471297:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829301.471309:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564381312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829301.471336:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829301/real 1787829301] req@ffff97688cb5e1c0 x1874670564380800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829301.471354:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564380800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829301.471365:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829301.471369:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829301.471373:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564380800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829301.471405:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564380928:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829301.471425:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829301.471436:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829301/real 1787829301] req@ffff97688cb5c780 x1874670564380928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829301.471444:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564380928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829301.471451:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829301.471454:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829301.471456:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564380928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829301.471460:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564381056:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829301.471472:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829301.471478:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829301/real 1787829301] req@ffff97688cb5c3c0 x1874670564381056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829301.471492:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564381056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829301.471498:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829301.471500:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829301.471502:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564381056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829301.471506:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564381184:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829301.471519:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829301.471525:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829301/real 1787829301] req@ffff97688cb5f0c0 x1874670564381184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829301.471533:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564381184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829301.471539:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829301.471541:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829301.471543:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564381184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829301.471546:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564381312:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829301.471557:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829301.471563:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829301/real 1787829301] req@ffff97688cb5d2c0 x1874670564381312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829301.471570:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564381312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829301.471580:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829301.471583:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829301.471585:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564381312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829301.726966:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829301.726966:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829301.727063:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829302.494974:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829302.750951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829302.750951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829302.750988:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829303.518934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829303.774913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829303.774914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829303.775011:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829304.542936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829304.798924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829304.798924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829304.798950:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829305.566909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829305.822911:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829305.822913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829305.822997:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829306.590957:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829306.590958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829306.590967:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829306.590973:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829306.590983:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4246 00000100:00080000:2.0:1787829306.590995:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829306.591003:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829306.591050:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564381440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829306.591076:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829306.591079:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829306.591080:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829306.591084:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4246 00000100:00080000:2.0:1787829306.591086:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829306.591090:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829306.591099:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829306.591106:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564381440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829306.591112:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564381568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829306.591120:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829306.591123:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829306.591125:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829306.591128:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4246 00000100:00080000:2.0:1787829306.591131:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829306.591133:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829306.591136:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829306.591144:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564381696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829306.591149:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829306.591152:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829306.591153:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829306.591172:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4246 00000100:00080000:2.0:1787829306.591175:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829306.591178:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829306.591183:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829306.591192:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564381824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829306.591199:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829306.591201:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829306.591203:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829306.591206:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4246 00000100:00080000:2.0:1787829306.591208:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829306.591211:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829306.591214:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829306.591221:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564381952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829306.591250:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564381568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829306.591281:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564381696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829306.591298:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564381824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829306.591312:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564381952:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829306.846942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829306.846946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829306.846969:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829307.614941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829307.870922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829307.870922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829307.870957:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829308.638902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829308.894939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829308.894939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829308.894977:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829309.662906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:0.0:1787829309.663092:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:0.0:1787829309.663146:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829309.663166:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787829309.663216:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787829309.663231:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787829309.663242:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:1.0:1787829309.663252:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829306/real 1787829309] req@ffff976889cfa940 x1874670564381568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787829309.663258:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:1.0:1787829309.663287:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564381568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829309.663303:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829309.663309:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829309.663313:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564381568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829309.663322:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829306/real 1787829309] req@ffff976889cfb480 x1874670564381696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829309.663335:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564381696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829309.663345:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829309.663349:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829309.663353:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564381696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829309.663360:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829306/real 1787829309] req@ffff976889cfad00 x1874670564381824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829309.663379:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564381824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829309.663387:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829309.663391:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829309.663395:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564381824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829309.663403:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829306/real 1787829309] req@ffff976889cfa1c0 x1874670564381952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829309.663415:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564381952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829309.663425:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829309.663429:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829309.663432:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564381952:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829309.663439:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829306/real 1787829309] req@ffff976889cf8f00 x1874670564381440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829309.663450:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564381440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829309.663459:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829309.663462:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829309.663465:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564381440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829309.918901:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829309.918926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829309.918967:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829310.686933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829310.942928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829310.942933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829310.942957:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829311.710955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829311.711017:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829311.711021:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829311.711025:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829311.711034:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4251 00000100:00080000:1.0:1787829311.711043:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829311.711051:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829311.711090:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564382080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829311.711120:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829311.711139:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829311.711142:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829311.711148:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4251 00000100:00080000:1.0:1787829311.711156:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829311.711161:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829311.711180:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564382080:192.168.203.139@tcp:250:kworker.0 00000008:00100000:1.0:1787829311.711181:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829311.711201:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564382208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829311.711214:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829311.711217:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829311.711219:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829311.711223:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4251 00000100:00080000:1.0:1787829311.711226:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829311.711231:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829311.711235:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829311.711245:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564382336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829311.711253:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829311.711256:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829311.711258:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829311.711263:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4251 00000100:00080000:1.0:1787829311.711285:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829311.711289:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829311.711295:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1787829311.711310:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829311.711312:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564382464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829311.711328:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829311.711332:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829311.711336:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829311.711343:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4251 00000100:00080000:1.0:1787829311.711350:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829311.711355:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829311.711361:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829311.711382:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829311/real 1787829311] req@ffff976889efed00 x1874670564382080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829311.711384:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564382592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829311.711479:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564382080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829311.711496:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829311.711502:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829311.711510:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564382080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829311.711552:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564382208:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829311.711584:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829311.711600:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829311/real 1787829311] req@ffff976889effc00 x1874670564382208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829311.711614:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564382208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829311.711625:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829311.711634:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829311.711638:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564382208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829311.711646:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564382336:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829311.711669:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829311.711682:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829311/real 1787829311] req@ffff976889efc3c0 x1874670564382336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829311.711694:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564382336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829311.711703:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829311.711707:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829311.711710:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564382336:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829311.711718:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564382464:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829311.711740:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829311.711753:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829311/real 1787829311] req@ffff976889efe940 x1874670564382464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829311.711767:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564382464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829311.711777:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829311.711780:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829311.711784:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564382464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829311.711791:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564382592:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829311.711813:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829311.711853:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829311/real 1787829311] req@ffff976889eff0c0 x1874670564382592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829311.711869:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564382592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829311.711878:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829311.711882:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829311.711886:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564382592:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829311.966941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829311.966941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829311.967110:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829312.734936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829312.990945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829312.990958:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829312.990972:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829313.758930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829314.014987:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829314.015030:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829314.015031:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829314.782919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829315.038899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829315.038937:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829315.038963:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829315.806949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829316.062922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829316.062923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829316.062947:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829316.830966:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829316.830974:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829316.830985:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829316.830990:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829316.831003:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4256 00000100:00080000:1.0:1787829316.831017:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829316.831026:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829316.831074:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564382720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829316.831119:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829316.831123:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829316.831126:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829316.831131:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4256 00000100:00080000:1.0:1787829316.831136:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829316.831142:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829316.831156:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829316.831164:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564382720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829316.831178:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564382848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829316.831193:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829316.831197:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829316.831200:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829316.831204:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4256 00000100:00080000:1.0:1787829316.831224:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829316.831229:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:0.0:1787829316.831237:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787829316.831246:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829316.831264:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564382976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829316.831276:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829316.831279:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829316.831281:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829316.831292:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4256 00000100:00100000:0.0:1787829316.831294:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829316/real 1787829316] req@ffff976889efcb40 x1874670564382720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829316.831302:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829316.831314:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829316.831326:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829316.831341:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564382720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829316.831344:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564383104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829316.831406:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829316.831409:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829316.831411:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829316.831417:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829316.831426:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:1.0:1787829316.831431:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4256 00000100:00100000:0.0:1787829316.831440:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564382720:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829316.831448:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829316.831458:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829316.831464:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829316.831477:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564382848:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829316.831484:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564383232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829316.831524:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829316.831546:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829316/real 1787829316] req@ffff976889efe1c0 x1874670564382848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829316.831557:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564382848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829316.831564:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829316.831567:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829316.831569:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564382848:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829316.831575:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564382976:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829316.831590:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829316.831597:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829316/real 1787829316] req@ffff976889efe580 x1874670564382976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829316.831605:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564382976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829316.831611:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829316.831613:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829316.831615:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564382976:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829316.831619:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564383104:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829316.831630:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829316.831637:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829316/real 1787829316] req@ffff976889eff480 x1874670564383104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829316.831645:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564383104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829316.831650:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829316.831652:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829316.831654:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564383104:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829316.831667:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564383232:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829316.831678:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829316.831684:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829316/real 1787829316] req@ffff976889efc780 x1874670564383232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829316.831692:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564383232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829316.831697:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829316.831698:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829316.831700:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564383232:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829317.086906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829317.086934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829317.086944:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829317.854884:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829318.110910:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829318.110912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829318.110940:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829318.878901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829319.134930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829319.134953:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829319.134962:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787829319.902994:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829320.158901:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829320.158910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829320.158929:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829320.926943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829321.182919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829321.182942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829321.182951:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829321.950951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829321.950966:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829321.950974:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829321.950980:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829321.950991:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4261 00000100:00080000:2.0:1787829321.951007:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829321.951016:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829321.951082:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564383360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829321.951114:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829321.951117:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829321.951119:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829321.951123:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4261 00000100:00080000:2.0:1787829321.951126:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829321.951130:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829321.951142:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829321.951157:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564383488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829321.951167:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829321.951171:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829321.951173:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829321.951180:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4261 00000100:00100000:0.0:1787829321.951182:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564383360:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829321.951184:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829321.951191:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829321.951203:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829321.951211:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564383616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829321.951218:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829321.951220:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829321.951221:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829321.951225:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4261 00000100:00080000:2.0:1787829321.951229:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829321.951232:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829321.951236:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829321.951244:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564383744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829321.951250:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829321.951252:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829321.951253:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829321.951256:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4261 00000100:00080000:2.0:1787829321.951259:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829321.951262:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829321.951265:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829321.951276:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564383872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829321.951283:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829321.951337:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829321/real 1787829321] req@ffff976889cf92c0 x1874670564383360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829321.951351:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564383360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829321.951360:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829321.951363:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829321.951366:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564383360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829321.951388:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564383488:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829321.951407:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829321.951416:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829321/real 1787829321] req@ffff976889cf8f00 x1874670564383488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829321.951425:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564383488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829321.951431:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829321.951433:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829321.951435:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564383488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829321.951440:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564383616:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829321.951451:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829321.951458:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829321/real 1787829321] req@ffff976889cfa1c0 x1874670564383616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829321.951465:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564383616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829321.951475:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829321.951477:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829321.951479:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564383616:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829321.951483:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564383744:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829321.951493:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829321.951499:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829321/real 1787829321] req@ffff976889cfad00 x1874670564383744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829321.951506:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564383744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829321.951511:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829321.951513:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829321.951515:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564383744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829321.951519:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564383872:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829321.951529:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829321.951534:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829321/real 1787829321] req@ffff976889cfb480 x1874670564383872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829321.951541:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564383872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829321.951546:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829321.951548:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829321.951550:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564383872:192.168.203.139@tcp:8:kworker.0 00010000:00010000:3.0:1787829322.206919:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829322.206970:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829322.206971:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829322.974929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829323.230919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829323.230919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829323.230986:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829323.998925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829324.254938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829324.254942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829324.255008:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829325.022893:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829325.278943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829325.278944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829325.278952:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829326.046906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829326.302948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829326.303007:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829326.303018:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829327.070906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829327.070969:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829327.070978:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829327.071000:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829327.071012:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4266 00000100:00080000:2.0:1787829327.071063:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829327.071072:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829327.071119:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564384000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829327.071145:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829327.071148:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829327.071150:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829327.071154:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4266 00000100:00080000:2.0:1787829327.071156:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829327.071159:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829327.071169:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829327.071180:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564384128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829327.071189:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829327.071193:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564384000:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829327.071193:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829327.071202:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829327.071206:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4266 00000100:00080000:2.0:1787829327.071209:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829327.071212:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829327.071216:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829327.071226:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564384256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829327.071234:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829327.071237:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829327.071238:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829327.071241:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4266 00000100:00080000:2.0:1787829327.071244:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829327.071247:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829327.071250:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829327.071258:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564384384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829327.071265:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829327.071268:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829327.071269:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829327.071275:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829327.071276:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4266 00000100:00080000:2.0:1787829327.071281:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829327.071284:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829327.071289:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829327.071298:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564384512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829327.071339:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829327/real 1787829327] req@ffff976889cfa940 x1874670564384000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829327.071360:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564384000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829327.071375:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829327.071381:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829327.071387:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564384000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829327.071418:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564384128:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829327.071449:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829327.071461:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829327/real 1787829327] req@ffff976889cf8000 x1874670564384128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829327.071471:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564384128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829327.071478:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829327.071481:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829327.071484:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564384128:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829327.071490:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564384256:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829327.071503:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829327.071510:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829327/real 1787829327] req@ffff976889cfb0c0 x1874670564384256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829327.071538:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564384256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829327.071545:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829327.071548:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829327.071551:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564384256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829327.071555:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564384384:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829327.071571:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829327.071579:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829327/real 1787829327] req@ffff976889cf9e00 x1874670564384384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829327.071588:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564384384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829327.071595:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829327.071598:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829327.071600:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564384384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829327.071605:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564384512:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829327.071632:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829327.071640:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829327/real 1787829327] req@ffff976889cfa580 x1874670564384512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829327.071649:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564384512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829327.071655:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829327.071657:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829327.071659:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564384512:192.168.203.139@tcp:8:kworker.0 00010000:00010000:3.0:1787829327.326933:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829327.326935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829327.326935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787829328.094896:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829328.350923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829328.350923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829328.350951:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829329.118901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829329.374909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829329.374911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829329.374934:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829330.142924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829330.398937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829330.398949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829330.398989:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829331.166958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829331.422961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829331.422961:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829331.422981:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829332.190922:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829332.190922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829332.190931:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829332.190936:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829332.190948:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4272 00000100:00080000:2.0:1787829332.190963:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829332.190972:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829332.191029:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564384640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829332.191060:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829332.191063:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829332.191064:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829332.191068:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4272 00000100:00080000:2.0:1787829332.191070:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829332.191073:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829332.191083:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829332.191095:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564384768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829332.191109:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829332.191111:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829332.191112:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829332.191115:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4272 00000100:00080000:2.0:1787829332.191118:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829332.191121:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829332.191126:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564384640:192.168.203.139@tcp:250:kworker.0 00000008:00100000:2.0:1787829332.191126:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829332.191138:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564384896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829332.191144:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829332.191146:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829332.191147:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829332.191151:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4272 00000100:00080000:2.0:1787829332.191153:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829332.191155:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829332.191159:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829332.191168:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564385024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829332.191174:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829332.191176:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829332.191177:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829332.191180:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4272 00000100:00080000:2.0:1787829332.191183:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829332.191186:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829332.191189:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829332.191198:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564385152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829332.191206:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829332.191261:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829332/real 1787829332] req@ffff976889cfbc00 x1874670564384640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829332.191279:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564384640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829332.191289:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829332.191293:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829332.191297:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564384640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829332.191322:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564384768:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829332.191338:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829332.191348:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829332/real 1787829332] req@ffff976889cfa580 x1874670564384768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829332.191357:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564384768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829332.191363:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829332.191365:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829332.191367:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564384768:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829332.191371:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564384896:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829332.191382:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829332.191388:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829332/real 1787829332] req@ffff976889cf9e00 x1874670564384896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829332.191395:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564384896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829332.191400:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829332.191402:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829332.191404:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564384896:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829332.191407:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564385024:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829332.191418:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829332.191431:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829332/real 1787829332] req@ffff976889cfb0c0 x1874670564385024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829332.191439:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564385024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829332.191445:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829332.191448:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829332.191450:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564385024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829332.191454:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564385152:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829332.191466:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829332.191472:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829332/real 1787829332] req@ffff976889cf8000 x1874670564385152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829332.191479:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564385152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829332.191484:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829332.191486:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829332.191488:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564385152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829332.446927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829332.446927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829332.446968:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829333.214943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829333.470959:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829333.470959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829333.471003:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829334.238911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829334.494904:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829334.494904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829334.494938:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829335.262920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829335.518905:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829335.518918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829335.518975:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829336.286914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829336.542942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829336.542942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829336.542985:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829337.310957:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829337.310957:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829337.310967:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829337.310971:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829337.310982:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4277 00000100:00080000:1.0:1787829337.310996:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829337.311005:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829337.311056:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564385280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829337.311083:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829337.311088:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829337.311090:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829337.311093:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4277 00000100:00080000:1.0:1787829337.311096:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829337.311099:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829337.311109:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829337.311122:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564385408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829337.311129:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564385280:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829337.311134:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829337.311138:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829337.311139:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829337.311142:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4277 00000100:00080000:1.0:1787829337.311144:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829337.311147:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829337.311150:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829337.311160:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564385536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829337.311166:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829337.311169:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829337.311170:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829337.311174:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4277 00000100:00080000:1.0:1787829337.311193:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829337.311196:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829337.311201:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1787829337.311205:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829337.311211:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564385664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829337.311220:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829337.311223:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829337.311224:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829337.311227:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4277 00000100:00080000:1.0:1787829337.311230:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829337.311232:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829337.311236:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829337.311244:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564385792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829337.311258:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829337/real 1787829337] req@ffff976889efc780 x1874670564385280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829337.311280:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564385280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829337.311293:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829337.311297:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829337.311300:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564385280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829337.311324:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564385408:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829337.311353:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829337.311364:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829337/real 1787829337] req@ffff976889eff480 x1874670564385408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829337.311373:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564385408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829337.311379:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829337.311381:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829337.311384:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564385408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829337.311393:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564385536:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829337.311406:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829337.311414:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829337/real 1787829337] req@ffff976889efe580 x1874670564385536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829337.311422:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564385536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829337.311427:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829337.311430:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829337.311432:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564385536:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829337.311436:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564385664:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829337.311448:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829337.311454:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829337/real 1787829337] req@ffff976889efe1c0 x1874670564385664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829337.311462:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564385664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829337.311467:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829337.311470:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829337.311472:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564385664:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829337.311475:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564385792:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829337.311485:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829337.311497:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829337/real 1787829337] req@ffff976889efcb40 x1874670564385792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829337.311503:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564385792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829337.311508:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829337.311511:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829337.311513:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564385792:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829337.566902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829337.566903:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829337.567037:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829338.334911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829338.590946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829338.590954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829338.591010:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829339.358961:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829339.614913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829339.614916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829339.614987:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829340.382913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829340.638940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829340.638940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829340.638977:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829341.406941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829341.662954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829341.662954:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829341.662987:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829342.430903:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829342.431062:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829342.431070:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829342.431076:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829342.431088:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4282 00000100:00080000:2.0:1787829342.431103:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829342.431113:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829342.431164:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564385920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829342.431194:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829342.431201:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829342.431203:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829342.431208:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4282 00000100:00080000:2.0:1787829342.431212:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829342.431215:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829342.431226:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829342.431236:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564386048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829342.431246:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829342.431248:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564385920:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829342.431249:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829342.431251:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829342.431257:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4282 00000100:00080000:2.0:1787829342.431260:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829342.431263:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829342.431283:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829342.431294:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564386176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829342.431301:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829342.431310:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829342.431311:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829342.431315:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4282 00000100:00080000:2.0:1787829342.431320:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829342.431323:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829342.431328:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829342.431353:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564386304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829342.431361:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829342.431363:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829342.431364:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829342.431370:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4282 00000800:00000100:0.0:1787829342.431371:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829342.431372:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829342.431377:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829342.431381:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829342.431388:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564386432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829342.431430:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829342/real 1787829342] req@ffff976889cfa940 x1874670564385920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829342.431446:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564385920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829342.431456:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829342.431459:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829342.431462:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564385920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829342.431489:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564386048:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829342.431507:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829342.431517:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829342/real 1787829342] req@ffff976889cfb480 x1874670564386048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829342.431525:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564386048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829342.431531:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829342.431533:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829342.431536:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564386048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829342.431539:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564386176:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829342.431549:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829342.431555:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829342/real 1787829342] req@ffff976889cfad00 x1874670564386176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829342.431563:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564386176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829342.431568:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829342.431570:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829342.431572:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564386176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829342.431575:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564386304:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829342.431600:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829342.431606:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829342/real 1787829342] req@ffff976889cfa1c0 x1874670564386304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829342.431614:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564386304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829342.431619:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829342.431622:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829342.431624:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564386304:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829342.431627:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564386432:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829342.431637:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829342.431643:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829342/real 1787829342] req@ffff976889cf8f00 x1874670564386432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829342.431650:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564386432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829342.431660:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829342.431662:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829342.431664:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564386432:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829342.686920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829342.686920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829342.686944:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829343.454974:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829343.710935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829343.710935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829343.710951:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829344.478879:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829344.734918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829344.734923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829344.734955:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829345.502950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829345.758953:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829345.758953:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829345.759005:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829346.526918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829346.782919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829346.782930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829346.782952:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829347.550942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829347.550998:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829347.551003:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829347.551007:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829347.551018:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4287 00000100:00080000:3.0:1787829347.551031:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829347.551039:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829347.551077:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564386560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829347.551100:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829347.551104:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829347.551105:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829347.551109:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4287 00000100:00080000:3.0:1787829347.551111:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829347.551114:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829347.551124:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829347.551142:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564386688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829347.551147:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564386560:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787829347.551153:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829347.551158:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829347.551160:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829347.551164:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4287 00000100:00080000:3.0:1787829347.551167:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829347.551171:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829347.551175:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829347.551187:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564386816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829347.551194:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829347.551198:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829347.551200:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829347.551203:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4287 00000100:00080000:3.0:1787829347.551207:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829347.551210:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829347.551213:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829347.551220:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564386944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829347.551226:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829347.551229:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829347.551231:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829347.551233:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829347.551269:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829347/real 1787829347] req@ffff97688cb5f840 x1874670564386560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829347.551283:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564386560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829347.551293:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829347.551296:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829347.551300:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564386560:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787829347.551309:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4287 00000100:00080000:3.0:1787829347.551313:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829347.551316:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829347.551321:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829347.551331:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564386688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829347.551334:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564387072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829347.551367:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829347.551393:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829347/real 1787829347] req@ffff97688cb5d680 x1874670564386688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829347.551403:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564386688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829347.551409:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829347.551412:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829347.551414:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564386688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829347.551419:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564386816:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829347.551436:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829347.551443:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829347/real 1787829347] req@ffff97688cb5de00 x1874670564386816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829347.551452:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564386816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829347.551459:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829347.551461:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829347.551463:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564386816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829347.551468:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564386944:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829347.551489:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829347.551499:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829347/real 1787829347] req@ffff97688cb5cb40 x1874670564386944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829347.551515:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564386944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829347.551521:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829347.551525:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829347.551528:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564386944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829347.551553:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564387072:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829347.551566:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829347.551574:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829347/real 1787829347] req@ffff97688cb5c000 x1874670564387072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829347.551583:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564387072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829347.551589:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829347.551591:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829347.551593:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564387072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829347.806943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829347.806957:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829347.806963:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829348.574914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829348.830913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829348.830922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829348.830937:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829349.598950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829349.854942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829349.854942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829349.854968:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829350.622917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829350.878961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829350.878961:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829350.878995:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829351.646934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829351.902941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829351.902945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829351.902973:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829352.670960:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829352.670967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829352.670970:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829352.670976:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829352.670988:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4292 00000100:00080000:2.0:1787829352.671001:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829352.671009:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829352.671054:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564387200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829352.671088:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829352.671092:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829352.671094:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829352.671100:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4292 00000100:00080000:2.0:1787829352.671105:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829352.671111:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829352.671124:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829352.671134:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564387200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829352.671145:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564387328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829352.671158:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829352.671162:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829352.671163:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829352.671166:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4292 00000100:00080000:2.0:1787829352.671170:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829352.671173:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829352.671178:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829352.671188:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564387456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829352.671195:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829352.671198:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829352.671199:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829352.671203:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4292 00000100:00080000:2.0:1787829352.671205:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829352.671207:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829352.671211:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829352.671216:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829352.671224:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564387584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829352.671233:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829352.671236:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829352.671238:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829352.671243:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4292 00000100:00080000:2.0:1787829352.671247:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829352.671251:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829352.671255:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829352/real 1787829352] req@ffff976889cf92c0 x1874670564387200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829352.671259:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829352.671276:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564387712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829352.671286:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564387200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829352.671302:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829352.671305:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829352.671309:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564387200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829352.671339:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564387328:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829352.671367:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829352.671383:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829352/real 1787829352] req@ffff976889cf8f00 x1874670564387328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829352.671396:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564387328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829352.671406:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829352.671410:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829352.671414:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564387328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829352.671421:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564387456:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829352.671442:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829352.671454:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829352/real 1787829352] req@ffff976889cfa1c0 x1874670564387456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829352.671467:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564387456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829352.671477:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829352.671480:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829352.671484:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564387456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829352.671492:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564387584:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829352.671513:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829352.671525:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829352/real 1787829352] req@ffff976889cfad00 x1874670564387584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829352.671538:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564387584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829352.671547:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829352.671551:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829352.671554:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564387584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829352.671560:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564387712:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829352.671582:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829352.671593:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829352/real 1787829352] req@ffff976889cfb480 x1874670564387712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829352.671607:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564387712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829352.671616:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829352.671620:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829352.671623:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564387712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829352.926969:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829352.926969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829352.926999:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829353.694926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829353.950933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829353.950935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829353.950961:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829354.718921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829354.974926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829354.974933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829354.974952:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829355.742901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829355.998874:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829355.998907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829355.998913:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829356.766901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829357.022910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829357.022913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829357.022967:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829357.790915:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829357.790924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829357.790925:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829357.790931:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829357.790942:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4297 00000100:00080000:3.0:1787829357.790954:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829357.790962:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829357.791009:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564387840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829357.791037:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829357.791040:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829357.791042:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829357.791045:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4297 00000100:00080000:3.0:1787829357.791047:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829357.791050:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829357.791059:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829357.791068:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564387968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829357.791089:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564387840:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787829357.791091:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829357.791097:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829357.791099:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829357.791102:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4297 00000100:00080000:3.0:1787829357.791104:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829357.791107:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829357.791110:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829357.791121:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564388096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829357.791127:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829357.791131:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829357.791132:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829357.791135:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4297 00000100:00080000:3.0:1787829357.791137:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829357.791140:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829357.791143:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829357.791150:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564388224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829357.791155:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829357.791158:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829357.791159:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829357.791162:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4297 00000100:00080000:3.0:1787829357.791165:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829357.791168:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829357.791171:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829357.791175:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829357.791182:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564388352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829357.791225:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829357/real 1787829357] req@ffff97688cb5c000 x1874670564387840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829357.791241:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564387840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829357.791251:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829357.791255:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829357.791260:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564387840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829357.791290:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564387968:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829357.791314:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829357.791324:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829357/real 1787829357] req@ffff97688cb5cb40 x1874670564387968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829357.791333:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564387968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829357.791340:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829357.791342:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829357.791345:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564387968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829357.791349:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564388096:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829357.791361:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829357.791367:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829357/real 1787829357] req@ffff97688cb5de00 x1874670564388096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829357.791375:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564388096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829357.791385:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829357.791387:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829357.791389:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564388096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829357.791392:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564388224:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829357.791404:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829357.791409:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829357/real 1787829357] req@ffff97688cb5d680 x1874670564388224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829357.791416:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564388224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829357.791422:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829357.791424:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829357.791426:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564388224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829357.791429:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564388352:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829357.791448:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829357.791460:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829357/real 1787829357] req@ffff97688cb5f840 x1874670564388352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829357.791473:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564388352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829357.791482:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829357.791485:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829357.791489:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564388352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829358.046906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829358.046913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829358.046950:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829358.814944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829359.070888:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829359.070902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829359.070995:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829359.838897:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829360.094898:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829360.094922:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829360.094925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829360.862937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829361.118886:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829361.118886:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829361.118944:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829361.886895:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829362.142944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829362.142944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829362.142972:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829362.910946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829362.910956:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829362.910964:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829362.910969:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829362.910979:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4302 00000100:00080000:1.0:1787829362.910990:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829362.910999:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829362.911037:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564388480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829362.911059:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829362.911062:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829362.911063:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829362.911066:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4302 00000100:00080000:1.0:1787829362.911069:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829362.911073:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829362.911081:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829362.911091:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564388608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829362.911098:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564388480:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829362.911100:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829362.911106:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829362.911109:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829362.911113:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4302 00000100:00080000:1.0:1787829362.911115:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829362.911117:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829362.911121:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829362.911130:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564388736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829362.911136:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829362.911138:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829362.911140:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829362.911143:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4302 00000100:00080000:1.0:1787829362.911160:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829362.911162:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829362.911166:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1787829362.911171:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829362.911178:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564388864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829362.911186:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829362.911189:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829362.911191:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829362.911193:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4302 00000100:00080000:1.0:1787829362.911195:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829362.911197:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829362.911201:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829362.911220:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829362/real 1787829362] req@ffff976889eff0c0 x1874670564388480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829362.911226:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564388992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829362.911280:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564388480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829362.911294:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829362.911298:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829362.911302:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564388480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829362.911327:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564388608:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829362.911351:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829362.911366:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829362/real 1787829362] req@ffff976889efe940 x1874670564388608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829362.911380:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564388608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829362.911390:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829362.911402:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829362.911406:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564388608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829362.911413:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564388736:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829362.911432:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829362.911442:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829362/real 1787829362] req@ffff976889efc3c0 x1874670564388736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829362.911455:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564388736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829362.911468:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829362.911470:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829362.911472:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564388736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829362.911476:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564388864:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829362.911490:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829362.911497:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829362/real 1787829362] req@ffff976889effc00 x1874670564388864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829362.911505:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564388864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829362.911510:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829362.911512:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829362.911514:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564388864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829362.911517:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564388992:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829362.911527:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829362.911533:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829362/real 1787829362] req@ffff976889efed00 x1874670564388992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829362.911540:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564388992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829362.911545:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829362.911547:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829362.911549:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564388992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829363.166937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829363.166937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829363.167005:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829363.934930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829364.190912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829364.190919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829364.190937:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829364.958938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829365.214935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829365.214938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829365.214998:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829365.982936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829366.238897:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829366.238927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829366.238928:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829367.006928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829367.262946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829367.262946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829367.262996:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829368.030910:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829368.030910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829368.030919:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829368.030923:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829368.030931:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4307 00000100:00080000:1.0:1787829368.030940:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829368.030946:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829368.030979:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564389120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829368.031001:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829368.031004:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829368.031005:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829368.031008:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4307 00000100:00080000:1.0:1787829368.031010:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829368.031012:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829368.031020:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829368.031029:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564389248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829368.031036:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829368.031039:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829368.031040:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829368.031044:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4307 00000100:00080000:1.0:1787829368.031047:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829368.031049:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564389120:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829368.031052:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829368.031059:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829368.031065:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564389376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829368.031072:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829368.031074:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829368.031076:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829368.031079:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4307 00000100:00080000:1.0:1787829368.031081:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829368.031083:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829368.031086:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829368.031093:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564389504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829368.031099:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829368.031102:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829368.031103:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829368.031105:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4307 00000100:00080000:1.0:1787829368.031109:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829368.031112:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787829368.031115:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829368.031119:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829368.031128:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564389632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829368.031151:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829368/real 1787829368] req@ffff976889efed00 x1874670564389120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829368.031165:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564389120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829368.031172:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829368.031175:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829368.031177:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564389120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829368.031183:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564389248:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829368.031196:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829368.031204:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829368/real 1787829368] req@ffff976889effc00 x1874670564389248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829368.031212:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564389248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829368.031217:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829368.031219:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829368.031222:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564389248:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829368.031241:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564389376:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829368.031252:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829368.031268:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829368/real 1787829368] req@ffff976889efc3c0 x1874670564389376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829368.031279:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564389376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829368.031285:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829368.031287:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829368.031289:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564389376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829368.031292:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564389504:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829368.031304:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829368.031310:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829368/real 1787829368] req@ffff976889efe940 x1874670564389504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829368.031318:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564389504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829368.031323:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829368.031324:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829368.031326:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564389504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829368.031330:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564389632:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829368.031339:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829368.031345:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829368/real 1787829368] req@ffff976889eff0c0 x1874670564389632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829368.031352:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564389632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829368.031358:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829368.031360:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829368.031363:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564389632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829368.286895:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829368.286895:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829368.286919:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829369.054985:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829369.310916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829369.310917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829369.310951:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829370.078895:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829370.334924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829370.334930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829370.334950:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829371.102901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829371.358922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829371.358922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829371.358984:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829372.126919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829372.382911:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829372.382911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829372.382937:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829373.150941:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829373.150945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829373.150953:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829373.150958:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829373.150967:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4313 00000100:00080000:3.0:1787829373.150978:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829373.150986:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829373.151030:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564389760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829373.151055:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829373.151057:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829373.151059:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829373.151062:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4313 00000100:00080000:3.0:1787829373.151065:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829373.151069:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829373.151077:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829373.151087:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564389888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829373.151101:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564389760:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787829373.151101:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829373.151106:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829373.151110:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829373.151114:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4313 00000100:00080000:3.0:1787829373.151117:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829373.151119:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829373.151122:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829373.151131:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564390016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829373.151138:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829373.151140:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829373.151142:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829373.151145:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4313 00000100:00080000:3.0:1787829373.151147:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829373.151149:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829373.151153:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829373.151159:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564390144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829373.151165:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829373.151168:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829373.151169:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829373.151171:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4313 00000100:00080000:3.0:1787829373.151174:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829373.151177:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829373.151180:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829373.151186:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564390272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829373.151232:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564389888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829373.151266:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564390016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829373.151296:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564390144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829373.151321:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564390272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829373.406925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829373.406925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829373.406974:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829374.174955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829374.430906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829374.430910:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829374.430938:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829375.198984:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829375.454951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829375.454953:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829375.455032:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829376.222983:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:2.0:1787829376.223201:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:2.0:1787829376.223251:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787829376.223274:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787829376.223326:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787829376.223343:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:1.0:1787829376.223346:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829373/real 1787829376] req@ffff97688cb5d2c0 x1874670564389760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787829376.223361:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:1.0:1787829376.223382:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564389760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787829376.223390:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00080000:1.0:1787829376.223407:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829376.223414:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829376.223420:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564389760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829376.223447:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829373/real 1787829376] req@ffff97688cb5f0c0 x1874670564389888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829376.223464:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564389888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829376.223475:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829376.223479:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829376.223483:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564389888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829376.223492:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829373/real 1787829376] req@ffff97688cb5c3c0 x1874670564390016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829376.223504:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564390016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829376.223519:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829376.223524:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829376.223529:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564390016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829376.223536:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829373/real 1787829376] req@ffff97688cb5c780 x1874670564390144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829376.223550:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564390144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829376.223558:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829376.223563:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829376.223566:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564390144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829376.223572:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829373/real 1787829376] req@ffff97688cb5e1c0 x1874670564390272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829376.223583:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564390272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829376.223590:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829376.223592:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829376.223596:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564390272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829376.478894:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829376.478944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829376.478973:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829377.246900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829377.502895:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829377.502899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829377.502947:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829378.270945:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829378.270946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829378.270958:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829378.270963:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829378.270974:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4318 00000100:00080000:3.0:1787829378.270985:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829378.270995:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829378.271031:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564390400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829378.271058:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829378.271063:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829378.271065:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829378.271069:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4318 00000100:00080000:3.0:1787829378.271079:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829378.271085:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829378.271094:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829378.271099:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564390400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829378.271110:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564390528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829378.271126:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829378.271129:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829378.271132:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829378.271136:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4318 00000100:00080000:3.0:1787829378.271140:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829378.271145:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829378.271151:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829378.271163:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564390656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829378.271173:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829378.271177:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829378.271181:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829378.271183:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787829378.271188:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4318 00000100:00080000:3.0:1787829378.271194:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829378.271199:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829378.271204:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829378.271216:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564390784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829378.271226:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829378.271232:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829378/real 1787829378] req@ffff97688cb5e1c0 x1874670564390400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829378.271233:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829378.271237:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829378.271245:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4318 00000100:00080000:3.0:1787829378.271253:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829378.271257:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829378.271263:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564390400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829378.271265:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829378.271279:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564390912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829378.271314:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829378.271322:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829378.271327:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564390400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829378.271356:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564390528:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829378.271383:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829378.271398:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829378/real 1787829378] req@ffff97688cb5c780 x1874670564390528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829378.271412:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564390528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829378.271421:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829378.271424:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829378.271428:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564390528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829378.271435:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564390656:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829378.271460:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829378.271473:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829378/real 1787829378] req@ffff97688cb5c3c0 x1874670564390656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829378.271486:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564390656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829378.271495:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829378.271499:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829378.271502:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564390656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829378.271508:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564390784:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829378.271528:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829378.271539:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829378/real 1787829378] req@ffff97688cb5f0c0 x1874670564390784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829378.271551:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564390784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829378.271559:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829378.271562:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829378.271565:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564390784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829378.271572:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564390912:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829378.271590:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829378.271600:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829378/real 1787829378] req@ffff97688cb5d2c0 x1874670564390912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829378.271618:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564390912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829378.271626:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829378.271629:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829378.271633:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564390912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829378.526898:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829378.526898:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829378.526946:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829379.294922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829379.550922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829379.550938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829379.550972:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829380.318971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829380.574943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829380.574943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829380.575020:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829381.342909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829381.598975:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829381.598975:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829381.599066:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829382.366944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829382.622933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829382.623001:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829382.623081:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00080000:3.0:1787829383.390900:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829383.390904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829383.390912:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829383.390917:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829383.390928:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4323 00000100:00080000:3.0:1787829383.390942:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829383.390952:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829383.391016:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564391040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829383.391051:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829383.391055:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829383.391057:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829383.391062:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4323 00000100:00080000:3.0:1787829383.391067:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829383.391072:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829383.391085:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829383.391100:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564391168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829383.391113:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564391040:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787829383.391139:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829383.391143:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829383.391146:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829383.391152:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4323 00000100:00080000:3.0:1787829383.391159:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829383.391163:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829383.391170:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829383.391185:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564391296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829383.391195:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829383.391196:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787829383.391206:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829383.391210:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829383.391217:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4323 00000100:00080000:3.0:1787829383.391223:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829383.391227:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829383.391232:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829383.391241:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829383/real 1787829383] req@ffff97688cb5f840 x1874670564391040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829383.391247:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564391424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829383.391269:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829383.391274:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829383.391277:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829383.391286:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564391040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829383.391286:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4323 00000100:00080000:3.0:1787829383.391298:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829383.391312:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:1.0:1787829383.391315:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829383.391316:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829383.391325:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829383.391332:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564391040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829383.391334:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564391552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829383.391376:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564391168:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829383.391408:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829383.391429:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829383/real 1787829383] req@ffff97688cb5d680 x1874670564391168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829383.391455:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564391168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829383.391462:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829383.391465:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829383.391468:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564391168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829383.391473:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564391296:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829383.391488:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829383.391495:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829383/real 1787829383] req@ffff97688cb5de00 x1874670564391296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829383.391503:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564391296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829383.391509:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829383.391513:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829383.391516:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564391296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829383.391521:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564391424:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829383.391538:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829383.391558:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829383/real 1787829383] req@ffff97688cb5cb40 x1874670564391424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829383.391576:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564391424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829383.391587:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829383.391590:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829383.391593:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564391424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829383.391602:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564391552:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829383.391638:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829383.391651:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829383/real 1787829383] req@ffff97688cb5c000 x1874670564391552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829383.391662:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564391552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829383.391668:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829383.391671:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829383.391675:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564391552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829383.646924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829383.646924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829383.647000:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829384.414935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829384.670954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829384.670960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829384.670983:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829385.438917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829385.694899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829385.694899:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829385.694964:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829386.462947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829386.718891:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829386.718898:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829386.718962:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829387.486894:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829387.742903:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829387.742903:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829387.742977:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829388.510904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829388.510929:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829388.510935:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829388.510939:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829388.510947:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4328 00000100:00080000:1.0:1787829388.510956:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829388.510962:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829388.511009:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564391680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829388.511046:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829388.511050:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829388.511052:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829388.511057:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4328 00000100:00080000:1.0:1787829388.511061:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829388.511066:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829388.511076:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829388.511086:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564391680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829388.511090:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564391808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829388.511100:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829388.511103:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829388.511105:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829388.511108:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4328 00000100:00080000:1.0:1787829388.511111:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829388.511115:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829388.511120:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829388.511129:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564391936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829388.511138:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829388.511141:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829388.511142:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829388.511147:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4328 00000100:00080000:1.0:1787829388.511149:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829388.511152:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787829388.511154:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829388.511158:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829388.511167:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564392064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829388.511175:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829388.511178:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829388.511179:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829388.511183:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4328 00000100:00080000:1.0:1787829388.511186:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829388.511189:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829388.511190:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829388/real 1787829388] req@ffff976889efcb40 x1874670564391680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829388.511193:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829388.511202:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564392192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829388.511208:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564391680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829388.511218:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829388.511222:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829388.511226:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564391680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829388.511249:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564391808:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829388.511271:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829388.511281:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829388/real 1787829388] req@ffff976889efe1c0 x1874670564391808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829388.511289:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564391808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829388.511296:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829388.511298:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829388.511301:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564391808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829388.511313:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564391936:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829388.511325:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829388.511332:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829388/real 1787829388] req@ffff976889efe580 x1874670564391936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829388.511340:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564391936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829388.511345:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829388.511347:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829388.511350:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564391936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829388.511353:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564392064:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829388.511363:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829388.511369:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829388/real 1787829388] req@ffff976889eff480 x1874670564392064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829388.511376:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564392064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829388.511388:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829388.511389:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829388.511392:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564392064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829388.511395:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564392192:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829388.511408:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829388.511415:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829388/real 1787829388] req@ffff976889efc780 x1874670564392192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829388.511422:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564392192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829388.511427:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829388.511429:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829388.511431:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564392192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829388.766928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829388.766928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829388.766989:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829389.534899:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829389.790931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829389.790932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829389.790957:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829390.558922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829390.814931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829390.814931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829390.815002:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829391.582909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829391.838902:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829391.838903:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829391.838961:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829392.606905:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829392.862928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829392.862928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829392.863099:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829393.630901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829393.630914:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829393.630920:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829393.630924:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829393.630933:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4333 00000100:00080000:3.0:1787829393.630942:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829393.630947:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829393.630981:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564392320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829393.631003:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829393.631006:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829393.631007:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829393.631010:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4333 00000100:00080000:3.0:1787829393.631012:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829393.631015:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829393.631023:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829393.631033:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564392448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829393.631038:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564392320:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787829393.631047:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829393.631051:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829393.631052:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829393.631055:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4333 00000100:00080000:3.0:1787829393.631057:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829393.631059:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829393.631062:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829393.631071:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564392576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829393.631078:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829393.631080:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829393.631081:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829393.631084:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4333 00000100:00080000:3.0:1787829393.631086:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829393.631089:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829393.631092:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829393.631100:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564392704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829393.631106:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787829393.631112:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829393.631115:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829393.631117:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829393.631120:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4333 00000100:00080000:3.0:1787829393.631123:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829393.631125:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829393.631129:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829393.631135:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564392832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829393.631160:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829393/real 1787829393] req@ffff97688cb5c000 x1874670564392320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829393.631181:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564392320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829393.631192:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829393.631196:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829393.631199:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564392320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829393.631219:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564392448:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829393.631238:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829393.631248:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829393/real 1787829393] req@ffff97688cb5cb40 x1874670564392448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829393.631256:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564392448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829393.631262:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829393.631265:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829393.631267:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564392448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829393.631270:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564392576:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829393.631281:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829393.631286:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829393/real 1787829393] req@ffff97688cb5de00 x1874670564392576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829393.631294:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564392576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829393.631299:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829393.631301:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829393.631303:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564392576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829393.631306:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564392704:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829393.631317:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829393.631322:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829393/real 1787829393] req@ffff97688cb5d680 x1874670564392704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829393.631334:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564392704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829393.631339:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829393.631341:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829393.631343:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564392704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829393.631346:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564392832:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829393.631355:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829393.631361:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829393/real 1787829393] req@ffff97688cb5f840 x1874670564392832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829393.631368:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564392832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829393.631373:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829393.631375:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829393.631377:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564392832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829393.886917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829393.886918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829393.886991:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829394.654914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829394.910954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829394.910965:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829394.910994:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829395.678936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829395.934908:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829395.934908:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829395.934996:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829396.702930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829396.958935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829396.958935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829396.959011:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829397.726895:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829397.982899:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829397.982900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829397.982964:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829398.750891:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829398.750901:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829398.750905:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829398.750914:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4338 00000100:00080000:3.0:1787829398.750923:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829398.750930:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829398.750958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787829398.750970:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564392960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829398.750999:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829398.751004:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829398.751006:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829398.751012:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4338 00000100:00080000:3.0:1787829398.751016:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829398.751019:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829398.751028:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829398.751035:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564392960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829398.751041:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564393088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829398.751051:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829398.751054:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829398.751055:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829398.751058:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4338 00000100:00080000:3.0:1787829398.751061:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829398.751064:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829398.751067:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829398.751075:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564393216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829398.751082:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829398.751084:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829398.751085:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829398.751090:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4338 00000800:00000100:0.0:1787829398.751092:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787829398.751094:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829398.751098:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829398.751102:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829398.751110:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564393344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829398.751117:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829398.751120:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829398.751121:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829398.751124:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4338 00000100:00080000:3.0:1787829398.751127:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829398.751130:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829398.751133:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829398/real 1787829398] req@ffff97688cb5d2c0 x1874670564392960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829398.751135:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829398.751152:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564392960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829398.751156:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564393472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829398.751169:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829398.751177:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829398.751183:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564392960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829398.751203:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564393088:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829398.751219:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829398.751228:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829398/real 1787829398] req@ffff97688cb5f0c0 x1874670564393088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829398.751236:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564393088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829398.751242:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829398.751244:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829398.751246:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564393088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829398.751249:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564393216:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829398.751269:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829398.751276:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829398/real 1787829398] req@ffff97688cb5c3c0 x1874670564393216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829398.751284:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564393216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829398.751293:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829398.751296:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829398.751298:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564393216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829398.751301:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564393344:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829398.751312:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829398.751318:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829398/real 1787829398] req@ffff97688cb5c780 x1874670564393344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829398.751324:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564393344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829398.751330:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829398.751332:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829398.751333:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564393344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829398.751337:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564393472:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829398.751352:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829398.751363:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829398/real 1787829398] req@ffff97688cb5e1c0 x1874670564393472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829398.751384:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564393472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829398.751393:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829398.751396:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829398.751400:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564393472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829399.006919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829399.006919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829399.006992:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829399.774886:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829400.030899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829400.030899:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829400.030946:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829400.798925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829401.054922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829401.054923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829401.054983:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829401.822936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829402.078929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829402.078929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829402.079002:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829402.846948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829403.102931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829403.102947:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829403.102986:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829403.870951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829403.870969:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829403.870978:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829403.870983:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829403.870994:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4343 00000100:00080000:1.0:1787829403.871006:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829403.871013:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829403.871053:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564393600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829403.871076:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829403.871079:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829403.871080:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829403.871083:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4343 00000100:00080000:1.0:1787829403.871086:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829403.871089:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829403.871097:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829403.871107:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564393728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829403.871118:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829403.871130:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829403.871133:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829403.871135:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564393600:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829403.871146:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4343 00000100:00080000:1.0:1787829403.871152:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829403.871157:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829403.871163:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829403.871174:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564393856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829403.871182:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829403.871185:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829403.871186:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829403.871189:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4343 00000100:00080000:1.0:1787829403.871194:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829403.871196:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829403.871200:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1787829403.871206:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829403.871216:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564393984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829403.871227:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829403.871231:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829403.871233:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829403.871238:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4343 00000100:00080000:1.0:1787829403.871242:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829403.871253:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829403.871260:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829403.871266:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829403/real 1787829403] req@ffff976889efc780 x1874670564393600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829403.871275:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564394112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829403.871300:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564393600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829403.871314:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829403.871319:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829403.871324:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564393600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829403.871333:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564393728:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829403.871356:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829403.871371:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829403/real 1787829403] req@ffff976889eff480 x1874670564393728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829403.871384:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564393728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829403.871394:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829403.871398:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829403.871402:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564393728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829403.871429:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564393856:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829403.871453:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829403.871466:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829403/real 1787829403] req@ffff976889efe580 x1874670564393856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829403.871479:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564393856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829403.871503:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829403.871507:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829403.871511:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564393856:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829403.871518:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564393984:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829403.871540:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829403.871552:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829403/real 1787829403] req@ffff976889efe1c0 x1874670564393984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829403.871565:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564393984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829403.871574:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829403.871578:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829403.871581:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564393984:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829403.871587:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564394112:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829403.871607:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829403.871620:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829403/real 1787829403] req@ffff976889efcb40 x1874670564394112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829403.871633:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564394112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829403.871652:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829403.871655:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829403.871659:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564394112:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829404.126919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829404.126952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829404.127011:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829404.894959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829405.150957:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829405.150968:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829405.150977:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829405.918943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829406.174963:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829406.174973:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829406.175031:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829406.942940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829407.198900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829407.198941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829407.198972:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829407.966964:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829408.223005:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829408.223008:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829408.223068:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829408.990938:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829408.990951:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1787829408.990952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829408.990956:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829408.990971:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4348 00000100:00080000:1.0:1787829408.990981:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829408.990987:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829408.991027:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564394240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829408.991058:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829408.991062:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829408.991064:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829408.991069:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4348 00000100:00080000:1.0:1787829408.991081:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829408.991086:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829408.991096:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829408.991103:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564394240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829408.991114:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564394368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829408.991124:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829408.991127:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829408.991129:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829408.991133:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4348 00000100:00080000:1.0:1787829408.991150:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829408.991153:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829408.991157:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829408.991159:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829408.991168:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564394496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829408.991177:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829408.991179:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829408.991180:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829408.991183:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4348 00000100:00080000:1.0:1787829408.991185:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829408.991187:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829408.991191:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829408.991199:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829408/real 1787829408] req@ffff976889eff0c0 x1874670564394240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829408.991202:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564394624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829408.991222:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829408.991225:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787829408.991228:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564394240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829408.991229:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829408.991237:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4348 00000100:00080000:1.0:1787829408.991242:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829408.991248:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829408.991248:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:0.0:1787829408.991257:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000008:00100000:1.0:1787829408.991259:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829408.991261:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564394240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829408.991276:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564394752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829408.991286:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564394368:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829408.991308:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829408.991319:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829408/real 1787829408] req@ffff976889efe940 x1874670564394368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829408.991328:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564394368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829408.991333:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829408.991335:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829408.991338:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564394368:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829408.991341:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564394496:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829408.991351:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829408.991356:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829408/real 1787829408] req@ffff976889efc3c0 x1874670564394496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829408.991363:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564394496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829408.991369:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829408.991371:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829408.991372:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564394496:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829408.991376:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564394624:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829408.991386:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829408.991391:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829408/real 1787829408] req@ffff976889effc00 x1874670564394624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829408.991403:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564394624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829408.991408:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829408.991409:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829408.991411:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564394624:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829408.991420:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564394752:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829408.991430:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829408.991436:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829408/real 1787829408] req@ffff976889efed00 x1874670564394752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829408.991443:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564394752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829408.991466:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829408.991469:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829408.991471:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564394752:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829409.246907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829409.246924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829409.246972:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829410.014975:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829410.270976:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829410.270983:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829410.271008:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829411.038940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829411.294946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829411.294972:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829411.295009:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829412.062959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829412.318943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829412.318944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829412.319011:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829413.086900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829413.342940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829413.342972:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829413.342982:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829414.110951:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829414.110976:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829414.110981:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829414.110993:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4354 00000100:00080000:2.0:1787829414.111003:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829414.111011:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829414.111024:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829414.111064:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564394880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829414.111093:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829414.111096:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829414.111098:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829414.111103:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4354 00000100:00080000:2.0:1787829414.111107:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829414.111111:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829414.111121:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829414.111136:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564395008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829414.111150:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564394880:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829414.111158:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829414.111163:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829414.111165:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829414.111169:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4354 00000100:00080000:2.0:1787829414.111174:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829414.111177:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829414.111183:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829414.111197:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564395136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829414.111207:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829414.111211:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829414.111213:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829414.111218:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4354 00000800:00000100:0.0:1787829414.111220:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829414.111224:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829414.111230:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829414.111235:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829414.111248:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564395264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829414.111258:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829414.111262:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829414.111264:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829414.111270:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829414/real 1787829414] req@ffff976889cfa940 x1874670564394880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829414.111270:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4354 00000100:00080000:2.0:1787829414.111276:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829414.111285:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829414.111290:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829414.111295:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564394880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829414.111304:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787829414.111306:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564395392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829414.111310:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829414.111317:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564394880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829414.111345:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564395008:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829414.111365:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829414.111374:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829414/real 1787829414] req@ffff976889cf8000 x1874670564395008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829414.111383:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564395008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829414.111389:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829414.111391:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829414.111402:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564395008:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829414.111406:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564395136:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829414.111435:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829414.111443:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829414/real 1787829414] req@ffff976889cfb0c0 x1874670564395136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829414.111451:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564395136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829414.111461:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829414.111463:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829414.111466:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564395136:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829414.111469:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564395264:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829414.111480:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829414.111486:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829414/real 1787829414] req@ffff976889cf9e00 x1874670564395264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829414.111500:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564395264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829414.111506:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829414.111508:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829414.111510:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564395264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829414.111513:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564395392:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829414.111523:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829414.111529:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829414/real 1787829414] req@ffff976889cfa580 x1874670564395392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829414.111536:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564395392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829414.111541:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829414.111543:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829414.111545:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564395392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829414.366926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829414.366926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829414.367028:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829415.134897:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829415.390899:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829415.390948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829415.390966:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829416.158933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829416.414928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829416.414928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829416.414978:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829417.182944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829417.438900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829417.438906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829417.438973:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829418.206942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829418.462912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829418.462927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829418.462938:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829419.230952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829419.230988:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829419.230996:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829419.231000:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829419.231010:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4359 00000100:00080000:0.0:1787829419.231021:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829419.231029:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829419.231066:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564395520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829419.231096:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829419.231101:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829419.231103:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829419.231107:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4359 00000100:00080000:0.0:1787829419.231111:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829419.231116:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829419.231127:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829419.231142:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564395648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829419.231153:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829419.231157:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829419.231159:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829419.231163:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4359 00000100:00100000:2.0:1787829419.231169:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564395520:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829419.231170:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829419.231179:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829419.231184:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829419.231196:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564395776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829419.231206:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829419.231209:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829419.231211:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829419.231215:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4359 00000100:00080000:0.0:1787829419.231219:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829419.231223:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829419.231227:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829419.231241:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564395904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829419.231242:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829419.231255:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829419.231259:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829419.231261:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829419.231265:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4359 00000100:00080000:0.0:1787829419.231269:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829419.231274:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829419.231279:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829419.231285:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829419/real 1787829419] req@ffff976882aee940 x1874670564395520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829419.231292:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564396032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829419.231321:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564395520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829419.231334:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829419.231339:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829419.231345:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564395520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829419.231353:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564395648:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829419.231398:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829419.231417:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829419/real 1787829419] req@ffff976882aec3c0 x1874670564395648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829419.231430:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564395648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829419.231440:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829419.231444:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829419.231451:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564395648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829419.231476:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564395776:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829419.231501:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829419.231520:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829419/real 1787829419] req@ffff976882aee580 x1874670564395776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829419.231535:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564395776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829419.231544:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829419.231548:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829419.231552:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564395776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829419.231558:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564395904:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829419.231580:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829419.231592:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829419/real 1787829419] req@ffff976882aed680 x1874670564395904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829419.231605:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564395904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829419.231621:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829419.231625:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829419.231628:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564395904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829419.231635:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564396032:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829419.231655:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829419.231668:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829419/real 1787829419] req@ffff976882aef0c0 x1874670564396032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829419.231680:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564396032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829419.231688:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829419.231691:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829419.231695:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564396032:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829419.486904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829419.486930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829419.486952:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829420.254899:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829420.510921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829420.510922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829420.510979:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829421.278951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829421.534935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829421.534935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829421.535000:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829422.302884:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829422.558922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829422.558924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829422.558973:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829423.326946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829423.582890:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829423.582900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829423.582938:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829424.350898:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829424.350898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829424.350910:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829424.350914:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829424.350922:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4364 00000100:00080000:0.0:1787829424.350931:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829424.350937:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829424.350968:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564396160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829424.350993:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829424.350996:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829424.350997:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829424.351000:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4364 00000100:00080000:0.0:1787829424.351002:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829424.351005:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829424.351013:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829424.351028:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564396288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829424.351034:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564396160:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829424.351048:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829424.351053:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829424.351056:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829424.351061:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4364 00000100:00080000:0.0:1787829424.351066:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829424.351070:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829424.351075:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829424.351089:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564396416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829424.351093:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829424.351107:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829424.351111:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829424.351113:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829424.351119:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4364 00000100:00080000:0.0:1787829424.351124:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829424.351129:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829424.351133:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829424/real 1787829424] req@ffff976882aeed00 x1874670564396160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829424.351136:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829424.351152:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564396544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829424.351160:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829424.351161:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564396160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829424.351164:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829424.351168:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829424.351175:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4364 00000100:00080000:0.0:1787829424.351182:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829424.351185:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:2.0:1787829424.351186:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829424.351188:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829424.351196:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829424.351200:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564396672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829424.351206:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564396160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829424.351237:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564396288:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829424.351262:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829424.351276:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829424/real 1787829424] req@ffff976882aef0c0 x1874670564396288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829424.351289:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564396288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829424.351298:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829424.351301:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829424.351305:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564396288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829424.351311:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564396416:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829424.351331:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829424.351364:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829424/real 1787829424] req@ffff976882aed680 x1874670564396416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829424.351386:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564396416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829424.351395:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829424.351398:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829424.351402:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564396416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829424.351408:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564396544:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829424.351429:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829424.351441:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829424/real 1787829424] req@ffff976882aee580 x1874670564396544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829424.351453:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564396544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829424.351462:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829424.351466:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829424.351470:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564396544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829424.351476:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564396672:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829424.351494:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829424.351506:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829424/real 1787829424] req@ffff976882aec3c0 x1874670564396672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829424.351518:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564396672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829424.351534:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829424.351538:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829424.351541:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564396672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829424.606907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829424.606913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829424.606958:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829425.374971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829425.630913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829425.630914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829425.630921:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829426.398934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829426.654917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829426.654917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829426.654963:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829427.422965:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829427.678897:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829427.678897:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829427.678943:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829428.446910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829428.702904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829428.702912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829428.702948:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829429.470927:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829429.470930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829429.470938:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829429.470944:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829429.470959:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4369 00000100:00080000:0.0:1787829429.470971:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829429.470979:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829429.471019:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564396800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829429.471050:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829429.471054:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829429.471056:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829429.471061:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4369 00000100:00080000:0.0:1787829429.471066:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829429.471071:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829429.471082:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829429.471096:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564396800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829429.471102:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564396928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829429.471116:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829429.471120:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829429.471123:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829429.471128:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4369 00000100:00080000:0.0:1787829429.471133:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829429.471137:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829429.471143:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829429.471155:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564397056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829429.471163:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829429.471168:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829429.471169:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829429.471172:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829429.471177:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4369 00000100:00080000:0.0:1787829429.471181:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829429.471185:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829429.471189:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829429.471197:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564397184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829429.471203:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829429.471205:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829429.471207:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829429.471210:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4369 00000100:00080000:0.0:1787829429.471213:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829429.471217:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829429.471219:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829429/real 1787829429] req@ffff976882aee940 x1874670564396800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829429.471220:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829429.471231:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564397312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829429.471249:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564396800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829429.471264:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829429.471269:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829429.471274:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564396800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829429.471300:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564396928:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829429.471348:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829429.471367:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829429/real 1787829429] req@ffff976882aec3c0 x1874670564396928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829429.471381:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564396928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829429.471391:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829429.471394:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829429.471398:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564396928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829429.471413:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564397056:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829429.471435:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829429.471448:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829429/real 1787829429] req@ffff976882aee580 x1874670564397056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829429.471461:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564397056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829429.471470:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829429.471473:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829429.471477:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564397056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829429.471484:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564397184:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829429.471505:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829429.471517:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829429/real 1787829429] req@ffff976882aed680 x1874670564397184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829429.471529:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564397184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829429.471539:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829429.471542:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829429.471546:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564397184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829429.471552:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564397312:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829429.471572:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829429.471584:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829429/real 1787829429] req@ffff976882aef0c0 x1874670564397312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829429.471596:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564397312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829429.471605:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829429.471608:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829429.471612:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564397312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829429.726901:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829429.726901:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829429.726960:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829430.494901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829430.750894:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829430.750903:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829430.750942:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829431.518931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829431.774927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829431.774927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829431.774994:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829432.542892:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829432.798923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829432.798923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829432.799018:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829433.566909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829433.822945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829433.823013:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829433.823049:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829434.590895:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829434.590913:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829434.590918:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829434.590926:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829434.590934:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4374 00000100:00080000:0.0:1787829434.590947:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829434.590953:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829434.590988:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564397440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829434.591009:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829434.591012:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829434.591013:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829434.591016:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4374 00000100:00080000:0.0:1787829434.591018:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829434.591021:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829434.591028:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829434.591038:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564397568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829434.591045:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829434.591049:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829434.591050:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829434.591053:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4374 00000100:00080000:0.0:1787829434.591055:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829434.591057:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829434.591061:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829434.591067:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564397440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829434.591069:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564397696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829434.591087:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829434.591089:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829434.591090:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829434.591094:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4374 00000100:00080000:0.0:1787829434.591095:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829434.591098:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829434.591101:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829434.591108:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564397824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829434.591114:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829434.591116:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829434.591117:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829434.591120:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4374 00000100:00080000:0.0:1787829434.591123:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829434.591125:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829434.591129:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829434.591135:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564397952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829434.591147:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829434.591198:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829434/real 1787829434] req@ffff976882aeed00 x1874670564397440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829434.591217:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564397440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829434.591227:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829434.591232:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829434.591236:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564397440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829434.591243:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564397568:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829434.591283:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829434.591298:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829434/real 1787829434] req@ffff976882aef0c0 x1874670564397568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829434.591311:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564397568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829434.591319:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829434.591323:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829434.591326:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564397568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829434.591352:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564397696:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829434.591375:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829434.591387:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829434/real 1787829434] req@ffff976882aed680 x1874670564397696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829434.591407:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564397696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829434.591416:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829434.591420:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829434.591423:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564397696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829434.591429:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564397824:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829434.591450:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829434.591462:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829434/real 1787829434] req@ffff976882aee580 x1874670564397824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829434.591473:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564397824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829434.591481:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829434.591485:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829434.591488:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564397824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829434.591494:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564397952:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829434.591513:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829434.591523:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829434/real 1787829434] req@ffff976882aec3c0 x1874670564397952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829434.591536:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564397952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829434.591552:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829434.591555:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829434.591558:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564397952:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829434.846926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829434.846926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829434.846984:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829435.614924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829435.870937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829435.870938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829435.870974:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829436.638974:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829436.894918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829436.894920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829436.895009:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829437.662898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829437.918938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829437.918938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829437.919048:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829438.686932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829438.942913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829438.942974:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829438.943002:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829439.710929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829439.710941:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829439.710949:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829439.710953:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829439.710963:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4379 00000100:00080000:3.0:1787829439.710974:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829439.710982:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829439.711027:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564398080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829439.711047:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829439.711050:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829439.711052:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829439.711055:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4379 00000100:00080000:3.0:1787829439.711058:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829439.711061:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829439.711070:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829439.711080:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564398208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829439.711088:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564398080:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787829439.711099:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829439.711102:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829439.711104:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829439.711107:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4379 00000100:00080000:3.0:1787829439.711109:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829439.711112:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829439.711116:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829439.711123:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564398336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829439.711130:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829439.711133:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829439.711134:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829439.711137:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4379 00000100:00080000:3.0:1787829439.711146:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829439.711148:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829439.711151:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829439.711159:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564398464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829439.711165:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829439.711168:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829439.711169:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829439.711172:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4379 00000100:00080000:3.0:1787829439.711174:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829439.711177:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829439.711180:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829439.711190:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564398592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829439.711205:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564398208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829439.711255:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564398336:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829439.711284:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564398464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829439.711306:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564398592:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829439.966920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829439.966930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829439.966987:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829440.734897:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829440.990921:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829440.990921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829440.990977:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829441.758894:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829442.014921:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829442.014921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829442.014984:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829442.782924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:1.0:1787829442.783117:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:1.0:1787829442.783154:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787829442.783167:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787829442.783204:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787829442.783231:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787829442.783242:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:2.0:1787829442.783254:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829439/real 1787829442] req@ffff97688cb5c780 x1874670564398208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787829442.783260:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:2.0:1787829442.783286:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564398208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829442.783299:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829442.783304:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829442.783308:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564398208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829442.783317:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829439/real 1787829442] req@ffff97688cb5c3c0 x1874670564398336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829442.783330:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564398336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829442.783340:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829442.783344:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829442.783347:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564398336:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829442.783354:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829439/real 1787829442] req@ffff97688cb5f0c0 x1874670564398464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829442.783381:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564398464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829442.783390:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829442.783394:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829442.783398:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564398464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829442.783407:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829439/real 1787829442] req@ffff97688cb5d2c0 x1874670564398592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829442.783419:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564398592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829442.783428:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829442.783431:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829442.783435:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564398592:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829442.783442:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829439/real 1787829442] req@ffff97688cb5e1c0 x1874670564398080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829442.783455:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564398080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829442.783464:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829442.783467:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829442.783471:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564398080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829443.038909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829443.038909:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829443.038984:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829443.806958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829444.062900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829444.062901:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829444.062975:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829444.830940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829444.830983:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829444.830991:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829444.830995:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829444.831005:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4384 00000100:00080000:0.0:1787829444.831014:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829444.831019:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829444.831048:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564398720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829444.831075:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829444.831080:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829444.831082:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829444.831087:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4384 00000100:00080000:0.0:1787829444.831091:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829444.831096:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829444.831104:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829444.831117:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564398848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829444.831122:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564398720:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829444.831127:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829444.831132:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829444.831134:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829444.831138:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4384 00000100:00080000:0.0:1787829444.831142:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829444.831146:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829444.831152:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829444.831166:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564398976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829444.831177:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829444.831181:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829444.831183:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829444.831186:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4384 00000100:00080000:0.0:1787829444.831191:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829444.831209:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829444.831218:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829444.831225:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829444.831240:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564399104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829444.831251:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829444.831256:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829444.831257:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829444/real 1787829444] req@ffff976882aee940 x1874670564398720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829444.831260:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829444.831270:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4384 00000100:00080000:0.0:1787829444.831290:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829444.831297:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564398720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829444.831298:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829444.831307:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829444.831317:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829444.831326:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564399232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829444.831326:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829444.831336:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564398720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829444.831374:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564398848:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829444.831402:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829444.831418:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829444/real 1787829444] req@ffff976882aec3c0 x1874670564398848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829444.831432:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564398848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829444.831442:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829444.831446:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829444.831450:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564398848:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829444.831456:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564398976:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829444.831479:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829444.831491:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829444/real 1787829444] req@ffff976882aee580 x1874670564398976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829444.831504:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564398976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829444.831519:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829444.831522:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829444.831526:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564398976:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829444.831533:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564399104:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829444.831555:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829444.831567:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829444/real 1787829444] req@ffff976882aed680 x1874670564399104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829444.831579:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564399104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829444.831588:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829444.831591:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829444.831594:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564399104:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829444.831600:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564399232:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829444.832260:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829444.832275:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829444/real 1787829444] req@ffff976882aef0c0 x1874670564399232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829444.832290:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564399232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829444.832298:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829444.832301:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829444.832303:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564399232:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829445.086937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829445.086937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829445.087001:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829445.854956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829446.110919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829446.110919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829446.110961:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829446.878968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829447.134935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829447.134935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829447.135002:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829447.902943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829448.158891:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829448.158951:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829448.158988:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829448.926970:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829449.182956:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829449.182956:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829449.183034:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829449.950941:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829449.950952:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829449.950956:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829449.950965:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4389 00000100:00080000:0.0:1787829449.950974:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829449.950981:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829449.951015:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564399360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829449.951016:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829449.951043:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829449.951046:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829449.951048:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829449.951051:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4389 00000100:00080000:0.0:1787829449.951054:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829449.951058:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829449.951066:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829449.951077:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564399488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829449.951084:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564399360:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829449.951089:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829449.951092:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829449.951093:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829449.951096:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4389 00000100:00080000:0.0:1787829449.951098:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829449.951100:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829449.951103:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829449.951111:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564399616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829449.951118:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829449.951120:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829449.951121:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829449.951124:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4389 00000100:00080000:0.0:1787829449.951127:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829449.951130:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829449.951134:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829449.951142:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564399744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829449.951146:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829449.951170:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829449.951174:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829449.951177:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829449.951182:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4389 00000100:00080000:0.0:1787829449.951188:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829449.951193:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829449.951198:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829449.951206:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829449/real 1787829449] req@ffff976882aeed00 x1874670564399360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829449.951214:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564399872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829449.951224:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564399360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829449.951235:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829449.951238:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829449.951242:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564399360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829449.951268:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564399488:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829449.951301:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829449.951313:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829449/real 1787829449] req@ffff976882aef0c0 x1874670564399488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829449.951322:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564399488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829449.951328:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829449.951330:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829449.951332:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564399488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829449.951337:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564399616:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829449.951349:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829449.951356:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829449/real 1787829449] req@ffff976882aed680 x1874670564399616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829449.951368:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564399616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829449.951373:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829449.951375:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829449.951376:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564399616:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829449.951380:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564399744:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829449.951391:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829449.951397:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829449/real 1787829449] req@ffff976882aee580 x1874670564399744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829449.951404:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564399744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829449.951409:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829449.951411:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829449.951420:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564399744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829449.951423:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564399872:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829449.951434:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829449.951441:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829449/real 1787829449] req@ffff976882aec3c0 x1874670564399872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829449.951448:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564399872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829449.951457:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829449.951459:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829449.951461:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564399872:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829450.206932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829450.206945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829450.206989:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829450.974957:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829451.230949:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829451.230949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829451.231075:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829451.998887:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829452.254918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829452.254918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829452.254977:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829453.022915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829453.278937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829453.278937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829453.278993:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829454.046901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829454.302921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829454.302929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829454.302959:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829455.070916:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829455.070923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829455.070928:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829455.070933:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829455.070942:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4394 00000100:00080000:3.0:1787829455.070955:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829455.070962:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829455.070991:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564400000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829455.071016:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829455.071019:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829455.071021:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829455.071024:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4394 00000100:00080000:3.0:1787829455.071026:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829455.071029:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829455.071037:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829455.071049:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564400000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829455.071050:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564400128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829455.071065:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829455.071068:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829455.071070:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829455.071073:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4394 00000100:00080000:3.0:1787829455.071076:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829455.071079:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829455.071083:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829455.071092:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564400256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829455.071099:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829455.071101:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829455.071103:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829455.071106:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4394 00000100:00080000:3.0:1787829455.071109:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829455.071114:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787829455.071116:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829455.071138:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829455.071149:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564400384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829455.071157:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829455.071159:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829455.071161:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829455.071165:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4394 00000100:00100000:0.0:1787829455.071170:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829455/real 1787829455] req@ffff97688cb5f840 x1874670564400000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829455.071170:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829455.071187:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564400000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829455.071195:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829455.071198:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829455.071200:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564400000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829455.071221:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564400128:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829455.071237:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787829455.071243:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829455.071249:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829455/real 1787829455] req@ffff97688cb5d680 x1874670564400128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829455.071253:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829455.071261:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564400128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829455.071269:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829455.071273:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829455.071274:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564400512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829455.071277:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564400128:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829455.071288:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564400256:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829455.071308:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829455.071318:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829455/real 1787829455] req@ffff97688cb5de00 x1874670564400256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829455.071327:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564400256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829455.071333:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829455.071335:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829455.071337:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564400256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829455.071341:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564400384:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829455.071351:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829455.071356:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829455/real 1787829455] req@ffff97688cb5cb40 x1874670564400384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829455.071364:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564400384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829455.071369:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829455.071371:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829455.071373:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564400384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829455.071385:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564400512:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829455.071395:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829455.071402:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829455/real 1787829455] req@ffff97688cb5c000 x1874670564400512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829455.071409:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564400512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829455.071414:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829455.071415:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829455.071417:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564400512:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829455.326946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829455.327013:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829455.327057:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829456.094963:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829456.350906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829456.350913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829456.350944:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829457.118916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829457.374925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829457.374925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829457.375014:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829458.142943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829458.398893:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829458.398895:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829458.398953:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829459.166899:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829459.422919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829459.422919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829459.422978:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829460.190953:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829460.190958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829460.190964:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829460.190969:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829460.190977:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4400 00000100:00080000:1.0:1787829460.190987:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829460.190993:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829460.191027:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564400640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829460.191048:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829460.191050:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829460.191052:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829460.191055:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4400 00000100:00080000:1.0:1787829460.191057:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829460.191059:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829460.191067:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829460.191079:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564400768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829460.191099:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564400640:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829460.191102:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829460.191108:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829460.191110:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829460.191113:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4400 00000100:00080000:1.0:1787829460.191116:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829460.191119:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829460.191124:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829460.191135:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564400896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829460.191143:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829460.191145:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829460.191147:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829460.191149:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4400 00000100:00080000:1.0:1787829460.191152:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829460.191154:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:0.0:1787829460.191158:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787829460.191159:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829460.191171:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564401024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829460.191178:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829460.191181:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829460.191182:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829460.191184:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4400 00000100:00080000:1.0:1787829460.191186:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829460.191189:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829460.191192:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829460.191201:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829460/real 1787829460] req@ffff976889efed00 x1874670564400640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829460.191202:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564401152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829460.191244:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564400640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829460.191257:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829460.191262:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829460.191266:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564400640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829460.191290:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564400768:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829460.191313:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829460.191327:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829460/real 1787829460] req@ffff976889effc00 x1874670564400768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829460.191339:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564400768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829460.191360:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829460.191364:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829460.191368:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564400768:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829460.191374:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564400896:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829460.191395:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829460.191406:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829460/real 1787829460] req@ffff976889efc3c0 x1874670564400896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829460.191419:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564400896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829460.191434:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829460.191438:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829460.191442:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564400896:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829460.191448:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564401024:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829460.191467:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829460.191476:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829460/real 1787829460] req@ffff976889efe940 x1874670564401024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829460.191484:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564401024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829460.191489:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829460.191492:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829460.191494:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564401024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829460.191498:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564401152:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829460.191509:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829460.191515:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829460/real 1787829460] req@ffff976889eff0c0 x1874670564401152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829460.191522:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564401152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829460.191527:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829460.191529:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829460.191531:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564401152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829460.446944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829460.446944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829460.447004:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829461.214902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829461.470928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829461.470928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829461.470955:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829462.239245:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829462.494917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829462.494918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829462.494992:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829463.262893:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829463.518919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829463.518919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829463.518967:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829464.286953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829464.542968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829464.542968:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829464.543044:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829465.310957:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829465.310966:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1787829465.310967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829465.310970:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829465.310981:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4405 00000100:00080000:0.0:1787829465.310991:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829465.310997:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829465.311031:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564401280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829465.311066:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829465.311068:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829465.311070:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829465.311073:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4405 00000100:00080000:0.0:1787829465.311076:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829465.311079:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829465.311088:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829465.311099:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564401408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829465.311108:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829465.311109:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564401280:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829465.311111:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829465.311116:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829465.311120:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4405 00000100:00080000:0.0:1787829465.311122:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829465.311125:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829465.311128:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829465.311136:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564401536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829465.311143:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829465.311145:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829465.311146:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829465.311149:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4405 00000100:00080000:0.0:1787829465.311151:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829465.311153:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829465.311156:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829465.311165:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564401664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829465.311172:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829465.311176:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829465.311179:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829465.311181:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829465.311185:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4405 00000100:00080000:0.0:1787829465.311188:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829465.311191:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829465.311194:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829465.311201:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564401792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829465.311223:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829465/real 1787829465] req@ffff976882aee940 x1874670564401280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829465.311254:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564401280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829465.311266:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829465.311270:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829465.311274:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564401280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829465.311298:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564401408:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829465.311318:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829465.311328:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829465/real 1787829465] req@ffff976882aec3c0 x1874670564401408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829465.311337:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564401408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829465.311343:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829465.311345:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829465.311348:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564401408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829465.311352:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564401536:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829465.311365:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829465.311372:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829465/real 1787829465] req@ffff976882aee580 x1874670564401536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829465.311380:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564401536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829465.311385:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829465.311387:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829465.311389:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564401536:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829465.311394:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564401664:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829465.311412:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829465.311424:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829465/real 1787829465] req@ffff976882aed680 x1874670564401664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829465.311437:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564401664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829465.311446:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829465.311450:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829465.311453:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564401664:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829465.311459:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564401792:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829465.311481:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829465.311493:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829465/real 1787829465] req@ffff976882aef0c0 x1874670564401792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829465.311512:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564401792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829465.311518:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829465.311520:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829465.311522:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564401792:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829465.566919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829465.566932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829465.566985:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829466.334979:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829466.590934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829466.590934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829466.591012:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829467.358918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829467.614897:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829467.614909:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829467.614956:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829468.382937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829468.638926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829468.638946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829468.639002:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829469.406989:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829469.662980:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829469.662986:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829469.663074:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829470.430961:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829470.430972:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829470.430981:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829470.430986:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829470.430997:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4410 00000100:00080000:3.0:1787829470.431009:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829470.431016:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829470.431076:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564401920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829470.431103:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829470.431106:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829470.431107:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829470.431111:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4410 00000100:00080000:3.0:1787829470.431113:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829470.431116:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829470.431126:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829470.431136:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564402048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829470.431147:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564401920:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787829470.431152:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829470.431157:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829470.431159:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829470.431162:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4410 00000100:00080000:3.0:1787829470.431164:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829470.431167:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829470.431170:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829470.431179:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564402176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829470.431184:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829470.431187:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829470.431188:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829470.431191:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4410 00000100:00080000:3.0:1787829470.431194:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829470.431196:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829470.431200:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829470.431207:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564402304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829470.431212:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829470.431214:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829470.431215:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829470.431218:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4410 00000100:00080000:3.0:1787829470.431221:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829470.431224:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829470.431227:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829470.431231:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829470.431238:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564402432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829470.431285:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829470/real 1787829470] req@ffff97688cb5c000 x1874670564401920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829470.431307:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564401920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829470.431320:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829470.431324:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829470.431329:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564401920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829470.431355:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564402048:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829470.431376:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829470.431389:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829470/real 1787829470] req@ffff97688cb5cb40 x1874670564402048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829470.431402:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564402048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829470.431412:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829470.431416:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829470.431420:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564402048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829470.431427:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564402176:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829470.431443:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829470.431453:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829470/real 1787829470] req@ffff97688cb5de00 x1874670564402176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829470.431465:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564402176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829470.431475:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829470.431478:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829470.431482:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564402176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829470.431487:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564402304:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829470.431505:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829470.431514:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829470/real 1787829470] req@ffff97688cb5d680 x1874670564402304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829470.431535:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564402304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829470.431545:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829470.431548:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829470.431551:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564402304:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829470.431558:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564402432:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829470.431577:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829470.431587:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829470/real 1787829470] req@ffff97688cb5f840 x1874670564402432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829470.431599:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564402432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829470.431608:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829470.431611:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829470.431615:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564402432:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829470.686975:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829470.686978:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829470.687046:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829471.454927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829471.710987:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829471.710988:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829471.711060:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829472.478932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829472.734992:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829472.734992:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829472.735086:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829473.502891:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829473.758908:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829473.758913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829473.759028:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829474.526900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829474.782922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829474.782929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829474.783002:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829475.550942:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829475.550945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829475.550953:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829475.550958:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829475.550966:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4415 00000100:00080000:3.0:1787829475.550976:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829475.551000:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829475.551042:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564402560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829475.551070:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829475.551073:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829475.551074:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829475.551077:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4415 00000100:00080000:3.0:1787829475.551080:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829475.551083:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829475.551092:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829475.551102:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564402688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829475.551110:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829475.551114:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829475.551115:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829475.551118:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4415 00000100:00080000:3.0:1787829475.551120:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829475.551125:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829475.551127:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564402560:192.168.203.139@tcp:250:kworker.0 00000008:00100000:3.0:1787829475.551129:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829475.551140:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564402816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829475.551148:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829475.551150:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829475.551151:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829475.551153:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4415 00000100:00080000:3.0:1787829475.551156:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829475.551158:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829475.551161:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829475.551169:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564402944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829475.551178:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829475.551182:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829475.551184:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829475.551187:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4415 00000100:00080000:3.0:1787829475.551191:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829475.551197:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787829475.551199:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829475.551208:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829475.551222:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564403072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829475.551248:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829475/real 1787829475] req@ffff97688cb5e1c0 x1874670564402560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829475.551269:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564402560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829475.551279:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829475.551284:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829475.551288:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564402560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829475.551296:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564402688:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829475.551323:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829475.551348:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829475/real 1787829475] req@ffff97688cb5d2c0 x1874670564402688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829475.551362:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564402688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829475.551372:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829475.551376:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829475.551379:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564402688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829475.551410:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564402816:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829475.551443:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829475.551453:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829475/real 1787829475] req@ffff97688cb5f0c0 x1874670564402816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829475.551462:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564402816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829475.551468:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829475.551470:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829475.551472:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564402816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829475.551476:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564402944:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829475.551496:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829475.551507:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829475/real 1787829475] req@ffff97688cb5c3c0 x1874670564402944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829475.551527:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564402944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829475.551536:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829475.551539:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829475.551543:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564402944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829475.551550:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564403072:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829475.551566:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829475.551574:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829475/real 1787829475] req@ffff97688cb5c780 x1874670564403072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829475.551581:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564403072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829475.551587:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829475.551589:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829475.551591:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564403072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829475.806926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829475.807024:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829475.807035:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829476.574923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829476.830959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829476.830969:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829476.830996:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829477.598944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829477.854925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829477.854926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829477.855007:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829478.622902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829478.878912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829478.878912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829478.878975:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829479.646976:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829479.902888:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829479.902932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829479.902995:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829480.670937:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829480.670958:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829480.670961:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829480.670970:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4420 00000100:00100000:2.0:1787829480.670978:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829480.670986:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829480.670993:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829480.671024:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564403200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829480.671044:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829480.671047:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829480.671049:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829480.671056:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4420 00000100:00080000:1.0:1787829480.671059:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829480.671062:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829480.671072:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829480.671084:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564403328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829480.671090:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564403200:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829480.671093:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829480.671096:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829480.671097:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829480.671100:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4420 00000100:00080000:1.0:1787829480.671103:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829480.671106:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829480.671110:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829480.671118:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564403456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829480.671125:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829480.671128:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829480.671129:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829480.671132:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4420 00000100:00080000:1.0:1787829480.671143:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829480.671147:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829480.671151:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1787829480.671154:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829480.671160:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564403584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829480.671167:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829480.671170:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829480.671171:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829480.671174:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4420 00000100:00080000:1.0:1787829480.671177:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829480.671179:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829480.671183:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829480.671191:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829480/real 1787829480] req@ffff976889efcb40 x1874670564403200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829480.671191:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564403712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829480.671206:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564403200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829480.671215:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829480.671219:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829480.671223:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564403200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829480.671243:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564403328:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829480.671260:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829480.671268:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829480/real 1787829480] req@ffff976889efe1c0 x1874670564403328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829480.671275:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564403328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829480.671281:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829480.671282:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829480.671284:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564403328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829480.671288:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564403456:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829480.671297:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829480.671303:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829480/real 1787829480] req@ffff976889efe580 x1874670564403456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829480.671309:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564403456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829480.671317:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829480.671319:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829480.671321:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564403456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829480.671324:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564403584:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829480.671332:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829480.671337:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829480/real 1787829480] req@ffff976889eff480 x1874670564403584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829480.671343:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564403584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829480.671347:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829480.671349:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829480.671350:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564403584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829480.671353:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564403712:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829480.671362:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829480.671366:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829480/real 1787829480] req@ffff976889efc780 x1874670564403712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829480.671372:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564403712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829480.671377:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829480.671378:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829480.671380:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564403712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829480.926884:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829480.926884:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829480.926935:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829481.694928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829481.950924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829481.950924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829481.950964:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829482.718887:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829482.974912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829482.974913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829482.974969:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829483.742899:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829483.998932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829483.998939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829483.998992:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829484.766909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829485.022893:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829485.022896:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829485.022953:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829485.790937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829485.790939:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829485.790946:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829485.790950:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829485.790959:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4425 00000100:00080000:0.0:1787829485.790969:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829485.790976:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829485.791009:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564403840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829485.791030:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829485.791033:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829485.791034:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829485.791036:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4425 00000100:00080000:0.0:1787829485.791039:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829485.791042:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829485.791049:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829485.791059:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564403840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829485.791060:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564403968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829485.791071:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829485.791073:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829485.791075:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829485.791077:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4425 00000100:00080000:0.0:1787829485.791079:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829485.791081:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829485.791084:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829485.791090:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564404096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829485.791095:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829485.791097:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829485.791098:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829485.791101:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4425 00000100:00080000:0.0:1787829485.791103:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829485.791105:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829485.791109:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787829485.791112:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829485.791119:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564404224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829485.791130:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829485.791132:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829485.791133:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829485.791136:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4425 00000100:00080000:0.0:1787829485.791138:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829485.791141:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829485.791144:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829485/real 1787829485] req@ffff976882aeed00 x1874670564403840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829485.791145:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829485.791158:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564404352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829485.791162:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564403840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829485.791173:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829485.791176:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829485.791179:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564403840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829485.791196:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564403968:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829485.791209:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829485.791216:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829485/real 1787829485] req@ffff976882aef0c0 x1874670564403968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829485.791223:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564403968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829485.791229:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829485.791231:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829485.791232:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564403968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829485.791235:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564404096:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829485.791244:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829485.791249:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829485/real 1787829485] req@ffff976882aed680 x1874670564404096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829485.791256:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564404096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829485.791267:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829485.791268:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829485.791270:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564404096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829485.791273:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564404224:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829485.791283:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829485.791288:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829485/real 1787829485] req@ffff976882aee580 x1874670564404224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829485.791295:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564404224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829485.791299:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829485.791301:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829485.791303:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564404224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829485.791305:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564404352:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829485.791319:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829485.791324:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829485/real 1787829485] req@ffff976882aec3c0 x1874670564404352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829485.791334:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564404352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829485.791339:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829485.791340:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829485.791342:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564404352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829486.046934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829486.046934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829486.046983:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829486.814948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829487.070937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829487.070937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829487.070971:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829487.838948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829488.094929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829488.094929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829488.094975:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829488.862921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829489.118900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829489.118907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829489.118955:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829489.886956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829490.142897:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829490.142905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829490.142930:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829490.910925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829490.910930:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829490.910938:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829490.910942:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829490.910951:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4430 00000100:00080000:0.0:1787829490.910961:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829490.910969:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829490.911002:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564404480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829490.911031:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829490.911035:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829490.911038:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829490.911043:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4430 00000100:00080000:0.0:1787829490.911047:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829490.911052:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829490.911057:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564404480:192.168.203.139@tcp:250:kworker.0 00000008:00100000:0.0:1787829490.911064:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829490.911082:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564404608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829490.911094:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829490.911099:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829490.911101:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829490.911106:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4430 00000800:00000100:1.0:1787829490.911108:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829490.911113:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829490.911120:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829490.911125:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829490.911139:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564404736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829490.911140:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829490/real 1787829490] req@ffff976882aee940 x1874670564404480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829490.911162:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564404480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829490.911173:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829490.911178:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829490.911183:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:0.0:1787829490.911185:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787829490.911188:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564404480:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829490.911189:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829490.911196:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4430 00000100:00080000:0.0:1787829490.911201:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829490.911205:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829490.911211:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829490.911212:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564404608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829490.911227:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564404864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829490.911239:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829490.911241:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829490.911245:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829490.911249:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829490.911254:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4430 00000100:00080000:0.0:1787829490.911258:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829490.911262:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829490/real 1787829490] req@ffff976882aec3c0 x1874670564404608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829490.911265:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829490.911274:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829490.911282:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564404608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829490.911289:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564404992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829490.911296:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829490.911304:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829490.911310:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564404608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829490.911317:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564404736:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829490.911340:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829490.911354:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829490/real 1787829490] req@ffff976882aee580 x1874670564404736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829490.911367:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564404736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829490.911380:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829490.911383:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829490.911388:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564404736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829490.911404:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564404864:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829490.911429:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829490.911443:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829490/real 1787829490] req@ffff976882aed680 x1874670564404864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829490.911458:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564404864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829490.911467:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829490.911471:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829490.911475:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564404864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829490.911481:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564404992:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829490.911499:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829490.911508:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829490/real 1787829490] req@ffff976882aef0c0 x1874670564404992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829490.911514:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564404992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829490.911519:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829490.911521:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829490.911523:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564404992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829491.166898:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829491.166900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829491.166942:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829491.934932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787829492.190926:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829492.190929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829492.190951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829492.958949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829493.214892:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829493.214910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829493.214922:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829493.982942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829494.238905:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829494.238912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829494.238930:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829495.006895:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829495.262889:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829495.262922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829495.262970:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829496.030920:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829496.030920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829496.030930:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829496.030934:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829496.030943:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4435 00000100:00080000:1.0:1787829496.030952:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829496.030957:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829496.030988:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564405120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829496.031011:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829496.031014:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829496.031015:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829496.031018:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4435 00000100:00080000:1.0:1787829496.031019:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829496.031022:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829496.031029:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829496.031037:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564405248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829496.031043:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829496.031047:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787829496.031049:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564405120:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829496.031050:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829496.031054:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4435 00000100:00080000:1.0:1787829496.031060:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829496.031062:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829496.031067:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829496.031078:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564405376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829496.031087:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829496.031090:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829496.031092:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829496.031096:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4435 00000100:00080000:1.0:1787829496.031100:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829496.031103:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787829496.031107:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829496.031111:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829496.031121:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564405504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829496.031127:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829496.031129:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829496.031131:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829496.031132:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4435 00000100:00080000:1.0:1787829496.031134:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829496.031137:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829496.031140:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829496.031142:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829496/real 1787829496] req@ffff976889efc780 x1874670564405120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829496.031149:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564405632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829496.031160:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564405120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829496.031168:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829496.031172:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829496.031174:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564405120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829496.031191:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564405248:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829496.031208:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829496.031220:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829496/real 1787829496] req@ffff976889eff480 x1874670564405248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829496.031232:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564405248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829496.031240:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829496.031244:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829496.031247:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564405248:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829496.031252:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564405376:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829496.031271:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829496.031289:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829496/real 1787829496] req@ffff976889efe580 x1874670564405376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829496.031301:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564405376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829496.031309:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829496.031311:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829496.031315:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564405376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829496.031320:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564405504:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829496.031336:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829496.031343:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829496/real 1787829496] req@ffff976889efe1c0 x1874670564405504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829496.031349:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564405504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829496.031355:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829496.031358:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829496.031360:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564405504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829496.031366:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564405632:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829496.031383:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829496.031392:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829496/real 1787829496] req@ffff976889efcb40 x1874670564405632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829496.031403:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564405632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829496.031410:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829496.031413:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829496.031416:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564405632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829496.286951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829496.286997:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829496.287025:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829497.054963:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829497.310915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829497.310916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829497.310947:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829498.078900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829498.334896:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829498.334896:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829498.334962:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829499.102960:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829499.358893:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829499.358935:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829499.358953:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829500.126885:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829500.382905:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829500.382913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829500.382963:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829501.150912:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829501.150913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829501.150922:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829501.150925:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829501.150932:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4441 00000100:00080000:1.0:1787829501.150941:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829501.150946:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829501.150973:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564405760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829501.151000:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829501.151003:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829501.151005:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829501.151009:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4441 00000100:00080000:1.0:1787829501.151012:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829501.151016:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829501.151024:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829501.151038:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564405888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829501.151041:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564405760:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829501.151055:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829501.151058:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829501.151060:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829501.151063:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4441 00000100:00080000:1.0:1787829501.151066:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829501.151068:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829501.151072:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829501.151080:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564406016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829501.151086:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829501.151088:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829501.151089:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829501.151092:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4441 00000100:00080000:1.0:1787829501.151094:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829501.151096:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829501.151099:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787829501.151106:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829501.151107:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564406144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829501.151121:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829501.151123:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829501.151125:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829501.151127:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4441 00000100:00080000:1.0:1787829501.151130:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829501.151132:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829501.151135:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829501.151142:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564406272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829501.151153:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829501/real 1787829501] req@ffff976889eff0c0 x1874670564405760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829501.151171:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564405760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829501.151180:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829501.151184:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829501.151188:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564405760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829501.151210:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564405888:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829501.151226:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829501.151234:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829501/real 1787829501] req@ffff976889efe940 x1874670564405888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829501.151242:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564405888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829501.151247:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829501.151248:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829501.151251:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564405888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829501.151256:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564406016:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829501.151272:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829501.151281:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829501/real 1787829501] req@ffff976889efc3c0 x1874670564406016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829501.151293:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564406016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829501.151300:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829501.151303:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829501.151305:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564406016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829501.151309:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564406144:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829501.151322:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829501.151330:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829501/real 1787829501] req@ffff976889effc00 x1874670564406144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829501.151341:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564406144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829501.151348:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829501.151351:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829501.151354:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564406144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829501.151359:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564406272:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829501.151376:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829501.151387:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829501/real 1787829501] req@ffff976889efed00 x1874670564406272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829501.151398:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564406272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829501.151403:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829501.151405:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829501.151407:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564406272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829501.406938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829501.406938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829501.407080:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829502.174882:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829502.430904:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829502.430917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829502.430928:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829503.198972:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829503.454939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829503.454981:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829503.455017:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829504.222882:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829504.478902:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829504.478902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829504.478963:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829505.247021:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829505.502895:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829505.502904:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829505.502998:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829506.270942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829506.270943:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829506.270952:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829506.270956:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829506.270966:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4446 00000100:00080000:2.0:1787829506.270977:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829506.270985:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829506.271026:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564406400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829506.272257:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829506.272262:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829506.272265:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829506.272273:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4446 00000100:00080000:2.0:1787829506.272283:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829506.272291:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829506.272301:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829506.272303:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564406400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829506.272332:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564406528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829506.272344:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829506.272347:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829506.272350:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829506.272354:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4446 00000100:00080000:2.0:1787829506.272366:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829506.272371:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829506.272376:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829506.272389:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564406656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829506.272400:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829506.272404:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829506.272406:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829506.272411:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564406528:192.168.203.139@tcp:38:kworker.0 00000100:00080000:2.0:1787829506.272414:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4446 00000100:00080000:2.0:1787829506.272422:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829506.272426:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829506.272432:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829506.272439:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564406656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829506.272447:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564406784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829506.272465:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829506.272469:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787829506.272469:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564406784:192.168.203.139@tcp:8:kworker.0 00000100:00080000:2.0:1787829506.272474:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829506.272483:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4446 00000100:00080000:2.0:1787829506.272490:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829506.272495:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829506.272502:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829506.272526:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564406912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829506.272569:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564406912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829506.526906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829506.526941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829506.526987:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829507.295031:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829507.550954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829507.550984:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829507.550990:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829508.318980:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829508.574892:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829508.574905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829508.574979:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829509.343090:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:2.0:1787829509.343104:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:2.0:1787829509.343147:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787829509.343163:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787829509.343197:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787829509.343204:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787829509.343215:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787829509.343225:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:3.0:1787829509.343295:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829506/real 1787829509] req@ffff976889cf8000 x1874670564406912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829509.343336:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564406912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829509.343351:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829509.343358:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829509.343362:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564406912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829509.343373:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829506/real 1787829509] req@ffff976889cfb0c0 x1874670564406784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829509.343386:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564406784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829509.343394:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829509.343398:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829509.343401:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564406784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829509.343409:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829506/real 1787829509] req@ffff976889cfa580 x1874670564406528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829509.343423:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564406528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829509.343432:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829509.343438:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829509.343441:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564406528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829509.343448:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829506/real 1787829509] req@ffff976889cf9e00 x1874670564406656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829509.343461:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564406656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829509.343469:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829509.343473:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829509.343476:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564406656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829509.343485:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829506/real 1787829509] req@ffff976889cfbc00 x1874670564406400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829509.343497:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564406400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829509.343504:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829509.343507:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829509.343510:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564406400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829509.598918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829509.598918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829509.598979:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829510.367002:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829510.622898:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829510.622901:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829510.622951:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829511.390914:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829511.390925:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829511.390929:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829511.390939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829511.390939:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4451 00000100:00080000:3.0:1787829511.390949:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829511.390955:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829511.390988:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564407040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829511.391011:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829511.391014:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829511.391016:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829511.391019:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4451 00000100:00080000:3.0:1787829511.391023:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829511.391027:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829511.391036:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829511.391037:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564407040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829511.391047:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564407168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829511.391055:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829511.391058:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829511.391059:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829511.391062:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4451 00000100:00080000:3.0:1787829511.391065:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829511.391067:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829511.391071:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829511.391079:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564407296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829511.391086:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829511.391089:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787829511.391089:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829511.391091:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829511.391095:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4451 00000100:00080000:3.0:1787829511.391097:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829511.391100:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829511.391104:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829511.391111:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564407424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829511.391119:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829511.391122:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829511.391123:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829511/real 1787829511] req@ffff97688cb5c780 x1874670564407040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829511.391124:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829511.391127:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4451 00000100:00080000:3.0:1787829511.391129:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829511.391132:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829511.391136:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829511.391138:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564407040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829511.391143:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564407552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829511.391148:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829511.391151:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829511.391154:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564407040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829511.391175:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564407168:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829511.391203:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829511.391216:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829511/real 1787829511] req@ffff97688cb5c3c0 x1874670564407168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829511.391228:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564407168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829511.391236:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829511.391239:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829511.391241:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564407168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829511.391246:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564407296:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829511.391264:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829511.391275:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829511/real 1787829511] req@ffff97688cb5f0c0 x1874670564407296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829511.391299:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564407296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829511.391305:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829511.391308:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829511.391311:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564407296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829511.391316:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564407424:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829511.391334:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829511.391343:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829511/real 1787829511] req@ffff97688cb5d2c0 x1874670564407424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829511.391358:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564407424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829511.391365:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829511.391368:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829511.391371:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564407424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829511.391375:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564407552:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829511.391391:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829511.391401:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829511/real 1787829511] req@ffff97688cb5e1c0 x1874670564407552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829511.391409:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564407552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829511.391415:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829511.391418:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829511.391421:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564407552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829511.646912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829511.646912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829511.646966:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829512.415020:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829512.670905:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829512.670911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829512.670915:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829513.438938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829513.694937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829513.694937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829513.694988:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829514.462941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829514.718953:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829514.718985:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829514.719070:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829515.486900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829515.742933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829515.742958:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829515.742979:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829516.510919:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829516.510922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829516.510929:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829516.510934:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829516.510941:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4456 00000100:00080000:3.0:1787829516.510949:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829516.510954:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829516.510982:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564407680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829516.511002:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829516.511005:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829516.511006:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829516.511008:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4456 00000100:00080000:3.0:1787829516.511011:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829516.511014:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829516.511021:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829516.511027:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564407680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829516.511031:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564407808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829516.511039:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829516.511041:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829516.511042:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829516.511045:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4456 00000100:00080000:3.0:1787829516.511046:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829516.511048:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829516.511051:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829516.511058:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564407936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829516.511063:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829516.511065:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829516.511066:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829516.511069:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4456 00000100:00080000:3.0:1787829516.511071:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829516.511074:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:1.0:1787829516.511078:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:3.0:1787829516.511080:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829516.511089:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564408064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829516.511095:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829516.511097:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829516.511098:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829516.511100:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4456 00000100:00080000:3.0:1787829516.511102:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829516.511105:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829516.511108:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829516.511111:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829516/real 1787829516] req@ffff97688cb5f840 x1874670564407680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829516.511117:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564408192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829516.511129:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564407680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829516.511137:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829516.511141:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829516.511144:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564407680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829516.511176:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564407808:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829516.511199:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829516.511210:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829516/real 1787829516] req@ffff97688cb5d680 x1874670564407808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829516.511222:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564407808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829516.511230:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829516.511233:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829516.511236:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564407808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829516.511242:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564407936:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829516.511260:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829516.511271:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829516/real 1787829516] req@ffff97688cb5de00 x1874670564407936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829516.511288:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564407936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829516.511296:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829516.511299:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829516.511302:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564407936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829516.511307:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564408064:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829516.511324:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829516.511334:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829516/real 1787829516] req@ffff97688cb5cb40 x1874670564408064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829516.511344:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564408064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829516.511352:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829516.511355:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829516.511358:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564408064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829516.511363:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564408192:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829516.511379:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829516.511388:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829516/real 1787829516] req@ffff97688cb5c000 x1874670564408192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829516.511398:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564408192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829516.511411:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829516.511414:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829516.511417:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564408192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829516.766905:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829516.766907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829516.766969:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829517.534938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829517.790955:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829517.790981:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829517.790987:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829518.558944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829518.814878:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829518.814903:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829518.814938:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829519.582912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829519.838894:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829519.838894:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829519.838936:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829520.606919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829520.862883:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829520.862889:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829520.862940:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829521.630877:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829521.630912:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829521.630917:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829521.630919:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829521.630926:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4461 00000100:00080000:1.0:1787829521.630934:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829521.630939:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829521.630973:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564408320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829521.630992:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829521.630994:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829521.630996:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829521.630998:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4461 00000100:00080000:1.0:1787829521.631000:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829521.631002:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829521.631009:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829521.631017:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564408448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829521.631024:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829521.631027:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829521.631028:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829521.631032:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564408320:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829521.631033:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4461 00000100:00080000:1.0:1787829521.631037:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829521.631041:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829521.631043:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829521.631051:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564408576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829521.631057:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829521.631059:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829521.631060:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829521.631064:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4461 00000100:00080000:1.0:1787829521.631068:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829521.631072:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829521.631076:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787829521.631083:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829521.631088:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564408704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829521.631099:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829521.631102:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829521.631104:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829521.631117:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4461 00000100:00080000:1.0:1787829521.631120:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829521.631123:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829521.631127:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829521.631132:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829521/real 1787829521] req@ffff976889efed00 x1874670564408320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829521.631137:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564408832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829521.631165:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564408320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829521.631176:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829521.631180:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829521.631183:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564408320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829521.631211:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564408448:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829521.631233:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829521.631245:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829521/real 1787829521] req@ffff976889effc00 x1874670564408448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829521.631256:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564408448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829521.631263:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829521.631266:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829521.631269:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564408448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829521.631275:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564408576:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829521.631297:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829521.631307:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829521/real 1787829521] req@ffff976889efc3c0 x1874670564408576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829521.631317:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564408576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829521.631325:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829521.631328:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829521.631331:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564408576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829521.631337:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564408704:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829521.631355:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829521.631365:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829521/real 1787829521] req@ffff976889efe940 x1874670564408704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829521.631382:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564408704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829521.631389:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829521.631392:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829521.631395:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564408704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829521.631401:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564408832:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829521.631419:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829521.631429:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829521/real 1787829521] req@ffff976889eff0c0 x1874670564408832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829521.631439:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564408832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829521.631446:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829521.631448:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829521.631451:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564408832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829521.886892:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829521.886905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829521.886951:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829522.654919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829522.910917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829522.910951:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829522.910995:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829523.678913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829523.934886:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829523.934899:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829523.934955:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829524.702920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829524.958963:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829524.959023:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829524.959106:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829525.726899:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829525.982904:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829525.982904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829525.983000:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829526.750886:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829526.750897:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829526.750903:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829526.750907:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829526.750915:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4466 00000100:00080000:1.0:1787829526.750924:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829526.750930:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829526.750959:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564408960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829526.750976:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829526.750978:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829526.750979:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829526.750982:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4466 00000100:00080000:1.0:1787829526.750984:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829526.750987:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829526.750995:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829526.751001:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564408960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829526.751004:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564409088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829526.751012:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829526.751014:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829526.751015:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829526.751018:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4466 00000100:00080000:1.0:1787829526.751020:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829526.751022:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829526.751026:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829526.751036:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564409216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829526.751046:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829526.751049:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829526.751051:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787829526.751054:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787829526.751058:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4466 00000100:00080000:1.0:1787829526.751063:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829526.751075:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829526.751081:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829526.751097:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564409344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829526.751106:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829526/real 1787829526] req@ffff976889efcb40 x1874670564408960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829526.751110:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829526.751118:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829526.751121:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829526.751126:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4466 00000100:00080000:1.0:1787829526.751129:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829526.751132:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564408960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829526.751138:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829526.751144:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787829526.751144:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829526.751153:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829526.751157:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564409472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829526.751162:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564408960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829526.751189:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564409088:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829526.751217:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829526.751228:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829526/real 1787829526] req@ffff976889efe1c0 x1874670564409088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829526.751239:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564409088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829526.751245:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829526.751247:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829526.751250:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564409088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829526.751254:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564409216:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829526.751268:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829526.751274:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829526/real 1787829526] req@ffff976889efe580 x1874670564409216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829526.751281:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564409216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829526.751286:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829526.751288:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829526.751289:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564409216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829526.751293:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564409344:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829526.751315:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829526.751329:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829526/real 1787829526] req@ffff976889eff480 x1874670564409344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829526.751341:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564409344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829526.751349:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829526.751353:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829526.751356:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564409344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829526.751363:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564409472:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829526.751388:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829526.751456:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829526/real 1787829526] req@ffff976889efc780 x1874670564409472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829526.751471:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564409472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829526.751481:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829526.751485:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829526.751488:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564409472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829527.006884:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829527.006929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829527.006942:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829527.774884:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829528.030996:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829528.031010:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829528.031044:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829528.798917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829529.054917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829529.054923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829529.054975:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829529.822917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829530.078909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829530.078919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829530.078944:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829530.846934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829531.102910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829531.102910:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829531.102972:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829531.870890:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829531.870898:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829531.870901:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829531.870906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829531.870909:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4471 00000100:00080000:3.0:1787829531.870918:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829531.870924:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829531.870952:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564409600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829531.870970:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829531.870973:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829531.870974:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829531.870976:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4471 00000100:00080000:3.0:1787829531.870978:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829531.870981:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829531.870987:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829531.870995:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564409728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829531.871002:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829531.871004:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829531.871006:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829531.871009:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4471 00000100:00080000:3.0:1787829531.871012:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829531.871013:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564409600:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787829531.871015:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829531.871021:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829531.871027:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564409856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829531.871037:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829531.871040:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829531.871042:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829531.871046:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4471 00000100:00080000:3.0:1787829531.871050:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829531.871053:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829531.871058:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829531.871070:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564409984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829531.871076:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787829531.871085:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829531.871088:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829531.871090:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829531.871093:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4471 00000100:00080000:3.0:1787829531.871097:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829531.871100:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829531.871104:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829531.871112:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564410112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829531.871129:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829531/real 1787829531] req@ffff97688cb5c000 x1874670564409600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829531.871144:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564409600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829531.871151:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829531.871153:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829531.871156:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564409600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829531.871172:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564409728:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829531.871187:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829531.871195:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829531/real 1787829531] req@ffff97688cb5cb40 x1874670564409728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829531.871202:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564409728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829531.871207:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829531.871209:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829531.871211:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564409728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829531.871214:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564409856:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829531.871224:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829531.871229:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829531/real 1787829531] req@ffff97688cb5de00 x1874670564409856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829531.871235:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564409856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829531.871239:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829531.871241:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829531.871243:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564409856:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829531.871246:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564409984:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829531.871254:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829531.871259:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829531/real 1787829531] req@ffff97688cb5d680 x1874670564409984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829531.871265:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564409984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829531.871273:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829531.871275:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829531.871277:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564409984:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829531.871280:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564410112:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829531.871287:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829531.871292:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829531/real 1787829531] req@ffff97688cb5f840 x1874670564410112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829531.871298:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564410112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829531.871302:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829531.871303:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829531.871305:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564410112:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829532.126888:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829532.126897:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829532.126949:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829532.894954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829533.150886:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829533.150906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829533.150952:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829533.918895:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829534.174924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829534.174925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829534.174984:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829534.942928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829535.198904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829535.198904:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829535.198935:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829535.966858:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829536.222896:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829536.222896:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829536.222951:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829536.990896:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829536.990907:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829536.990913:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829536.990917:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829536.990925:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4476 00000100:00080000:0.0:1787829536.990935:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829536.990941:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829536.990977:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564410240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829536.991007:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829536.991014:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829536.991016:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829536.991020:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4476 00000100:00080000:0.0:1787829536.991024:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829536.991029:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829536.991039:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829536.991047:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564410240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829536.991053:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564410368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829536.991061:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829536.991064:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829536.991067:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829536.991071:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4476 00000100:00080000:0.0:1787829536.991075:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829536.991079:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829536.991084:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829536.991095:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564410496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829536.991100:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829536.991110:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829536.991112:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829536.991114:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829536.991118:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4476 00000100:00080000:0.0:1787829536.991121:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829536.991123:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829536.991128:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829536.991133:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829536/real 1787829536] req@ffff976882aeed00 x1874670564410240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829536.991139:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564410624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829536.991162:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829536.991165:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829536.991166:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829536.991169:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4476 00000100:00080000:0.0:1787829536.991172:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829536.991172:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564410240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829536.991178:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829536.991183:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829536.991189:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829536.991193:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564410752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829536.991198:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829536.991206:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564410240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829536.991230:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564410368:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829536.991249:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829536.991261:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829536/real 1787829536] req@ffff976882aef0c0 x1874670564410368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829536.991273:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564410368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829536.991281:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829536.991284:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829536.991287:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564410368:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829536.991293:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564410496:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829536.991307:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829536.991313:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829536/real 1787829536] req@ffff976882aed680 x1874670564410496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829536.991320:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564410496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829536.991329:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829536.991331:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829536.991333:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564410496:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829536.991336:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564410624:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829536.991350:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829536.991359:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829536/real 1787829536] req@ffff976882aee580 x1874670564410624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829536.991370:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564410624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829536.991378:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829536.991381:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829536.991384:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564410624:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829536.991389:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564410752:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829536.991407:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829536.991414:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829536/real 1787829536] req@ffff976882aec3c0 x1874670564410752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829536.991420:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564410752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829536.991425:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829536.991426:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829536.991429:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564410752:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829537.246901:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829537.246901:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829537.246958:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829538.014920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829538.270888:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829538.270920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829538.270930:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829539.038898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829539.294935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829539.294935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829539.295020:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829540.062933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829540.318906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829540.318947:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829540.319024:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829541.086900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829541.342941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829541.342941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829541.342993:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829542.110913:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829542.110920:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829542.110924:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829542.110929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829542.110931:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4482 00000100:00080000:1.0:1787829542.110942:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829542.110948:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829542.110983:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564410880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829542.111008:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829542.111011:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829542.111013:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829542.111017:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4482 00000100:00080000:1.0:1787829542.111020:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829542.111024:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829542.111033:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829542.111043:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564411008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829542.111057:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564410880:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829542.111063:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829542.111068:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829542.111069:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829542.111074:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4482 00000100:00080000:1.0:1787829542.111078:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829542.111082:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829542.111088:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829542.111100:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564411136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829542.111109:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829542.111114:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1787829542.111115:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787829542.111118:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829542.111125:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4482 00000100:00080000:1.0:1787829542.111130:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829542.111134:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829542.111140:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829542.111150:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829542/real 1787829542] req@ffff976889efc780 x1874670564410880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829542.111153:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564411264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829542.111175:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829542.111181:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829542.111182:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829542.111185:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4482 00000100:00100000:3.0:1787829542.111189:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564410880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829542.111191:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829542.111197:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829542.111203:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787829542.111204:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829542.111209:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829542.111213:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564411392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829542.111214:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564410880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829542.111238:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564411008:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829542.111254:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829542.111262:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829542/real 1787829542] req@ffff976889eff480 x1874670564411008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829542.111270:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564411008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829542.111275:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829542.111277:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829542.111279:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564411008:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829542.111282:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564411136:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829542.111293:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829542.111298:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829542/real 1787829542] req@ffff976889efe580 x1874670564411136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829542.111305:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564411136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829542.111314:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829542.111316:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829542.111319:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564411136:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829542.111321:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564411264:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829542.111330:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829542.111335:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829542/real 1787829542] req@ffff976889efe1c0 x1874670564411264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829542.111341:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564411264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829542.111345:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829542.111347:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829542.111349:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564411264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829542.111351:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564411392:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829542.111366:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829542.111372:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829542/real 1787829542] req@ffff976889efcb40 x1874670564411392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829542.111378:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564411392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829542.111382:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829542.111384:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829542.111386:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564411392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829542.366923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829542.366923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829542.366991:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829543.134884:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829543.390902:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829543.390912:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829543.390955:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829544.158905:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829544.414916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829544.414916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829544.414973:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829545.182890:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829545.438913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829545.438916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829545.438978:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829546.206929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829546.462915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829546.462915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829546.462980:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829547.230918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829547.230933:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829547.230941:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829547.230945:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829547.230955:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4487 00000100:00080000:3.0:1787829547.230966:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829547.230975:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829547.231011:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564411520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829547.231040:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829547.231043:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829547.231045:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829547.231050:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4487 00000100:00080000:3.0:1787829547.231054:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829547.231059:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829547.231069:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564411520:192.168.203.139@tcp:250:kworker.0 00000008:00100000:3.0:1787829547.231069:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829547.231086:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564411648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829547.231096:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829547.231099:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829547.231101:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829547.231104:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4487 00000100:00080000:3.0:1787829547.231108:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829547.231113:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829547.231118:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829547.231127:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829547.231131:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564411776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829547.231143:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829547.231147:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829547.231150:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829547.231154:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4487 00000100:00080000:3.0:1787829547.231158:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829547.231163:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829547/real 1787829547] req@ffff97688cb5e1c0 x1874670564411520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829547.231164:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829547.231173:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829547.231182:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564411520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829547.231188:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564411904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829547.231193:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829547.231202:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:3.0:1787829547.231204:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829547.231207:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564411520:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787829547.231210:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829547.231215:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829547.231219:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4487 00000100:00080000:3.0:1787829547.231222:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829547.231226:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829547.231230:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829547.231235:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564411648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829547.231245:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564412032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829547.231255:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829547.231269:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829547/real 1787829547] req@ffff97688cb5d2c0 x1874670564411648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829547.231287:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564411648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829547.231296:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829547.231299:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829547.231303:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564411648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829547.231309:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564411776:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829547.231330:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829547.231342:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829547/real 1787829547] req@ffff97688cb5f0c0 x1874670564411776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829547.231352:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564411776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829547.231357:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829547.231360:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829547.231362:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564411776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829547.231366:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564411904:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829547.231380:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829547.231387:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829547/real 1787829547] req@ffff97688cb5c3c0 x1874670564411904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829547.231394:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564411904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829547.231403:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829547.231405:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829547.231407:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564411904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829547.231422:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564412032:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829547.231432:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829547.231438:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829547/real 1787829547] req@ffff97688cb5c780 x1874670564412032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829547.231445:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564412032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829547.231450:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829547.231452:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829547.231454:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564412032:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829547.486901:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829547.486901:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829547.486944:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829548.254953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829548.510942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829548.511056:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829548.511115:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829549.278887:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829549.534942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829549.534946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829549.534980:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829550.302918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829550.558909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829550.558917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829550.558963:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829551.326904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829551.582900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829551.582917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829551.582971:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829552.350916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829552.350931:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829552.350937:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829552.350941:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829552.350950:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4492 00000100:00080000:0.0:1787829552.350958:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829552.350964:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829552.350999:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564412160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829552.351027:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829552.351034:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829552.351036:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829552.351040:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4492 00000100:00080000:0.0:1787829552.351045:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829552.351050:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829552.351059:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829552.351072:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564412288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829552.351076:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564412160:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829552.351091:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829552.351094:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829552.351095:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829552.351098:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4492 00000100:00080000:0.0:1787829552.351101:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829552.351103:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829552.351106:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829552.351115:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564412416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829552.351129:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829552.351131:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829552.351133:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829552.351136:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4492 00000100:00080000:0.0:1787829552.351139:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829552.351143:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:1.0:1787829552.351147:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:0.0:1787829552.351150:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829552.351161:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564412544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829552.351171:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829552.351174:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829552.351176:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829552.351181:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4492 00000100:00080000:0.0:1787829552.351185:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829552.351188:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829552/real 1787829552] req@ffff976882aee940 x1874670564412160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829552.351194:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829552.351205:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829552.351214:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564412160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829552.351222:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564412672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829552.351225:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829552.351232:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829552.351239:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564412160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829552.351260:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564412288:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829552.351276:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829552.351285:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829552/real 1787829552] req@ffff976882aec3c0 x1874670564412288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829552.351293:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564412288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829552.351299:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829552.351301:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829552.351303:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564412288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829552.351307:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564412416:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829552.351316:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829552.351322:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829552/real 1787829552] req@ffff976882aee580 x1874670564412416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829552.351330:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564412416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829552.351339:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829552.351341:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829552.351343:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564412416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829552.351346:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564412544:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829552.351356:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829552.351361:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829552/real 1787829552] req@ffff976882aed680 x1874670564412544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829552.351368:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564412544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829552.351373:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829552.351375:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829552.351377:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564412544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829552.351381:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564412672:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829552.351390:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829552.351395:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829552/real 1787829552] req@ffff976882aef0c0 x1874670564412672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829552.351403:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564412672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829552.351407:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829552.351409:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829552.351411:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564412672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829552.606926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829552.606928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829552.606985:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829553.374907:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829553.630924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829553.630928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829553.630971:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829554.398903:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829554.654916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829554.654916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829554.654976:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829555.422950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829555.678904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829555.678904:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829555.678938:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829556.446946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829556.702906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829556.702908:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829556.702963:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829557.470950:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829557.470950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829557.470970:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829557.470978:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829557.470995:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4497 00000100:00080000:0.0:1787829557.471010:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829557.471018:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829557.471063:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564412800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829557.471097:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829557.471100:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829557.471102:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829557.471105:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4497 00000100:00080000:0.0:1787829557.471107:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829557.471110:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829557.471120:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829557.471133:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564412928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829557.471142:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829557.471145:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829557.471147:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829557.471151:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4497 00000100:00100000:1.0:1787829557.471157:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564412800:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829557.471161:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829557.471175:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829557.471181:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829557.471202:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564413056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829557.471212:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829557.471216:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829557.471218:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829557.471223:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4497 00000100:00080000:0.0:1787829557.471237:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829557.471246:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:1.0:1787829557.471249:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:0.0:1787829557.471255:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829557.471273:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564413184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829557.471283:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829557.471286:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829557.471289:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829557.471293:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4497 00000100:00080000:0.0:1787829557.471298:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829557.471307:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829557.471308:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829557/real 1787829557] req@ffff976882aeed00 x1874670564412800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829557.471313:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829557.471335:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564413312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829557.471344:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564412800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829557.471364:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829557.471369:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829557.471375:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564412800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829557.471404:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564412928:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829557.471446:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829557.471461:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829557/real 1787829557] req@ffff976882aef0c0 x1874670564412928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829557.471475:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564412928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829557.471485:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829557.471488:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829557.471493:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564412928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829557.471500:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564413056:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829557.471523:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829557.471535:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829557/real 1787829557] req@ffff976882aed680 x1874670564413056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829557.471558:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564413056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829557.471567:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829557.471570:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829557.471574:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564413056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829557.471581:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564413184:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829557.471603:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829557.471616:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829557/real 1787829557] req@ffff976882aee580 x1874670564413184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829557.471629:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564413184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829557.471638:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829557.471642:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829557.471646:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564413184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829557.471653:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564413312:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829557.471674:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829557.471698:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829557/real 1787829557] req@ffff976882aec3c0 x1874670564413312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829557.471712:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564413312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829557.471720:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829557.471724:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829557.471727:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564413312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829557.726905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829557.726907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829557.726939:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829558.494924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829558.750895:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829558.750912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829558.750976:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829559.518913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829559.774877:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829559.774904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829559.774935:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829560.542887:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829560.798898:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829560.798898:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829560.798940:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829561.566926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829561.822920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829561.822925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829561.823114:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829562.590880:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829562.590888:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829562.590893:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829562.590901:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4502 00000100:00080000:0.0:1787829562.590911:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829562.590917:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829562.590945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829562.590954:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564413440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829562.590980:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829562.590984:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829562.590987:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829562.590993:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4502 00000100:00080000:0.0:1787829562.590998:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829562.591003:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829562.591013:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564413440:192.168.203.139@tcp:250:kworker.0 00000008:00100000:0.0:1787829562.591015:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829562.591036:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564413568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829562.591051:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829562.591056:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829562.591058:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829562.591061:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4502 00000100:00080000:0.0:1787829562.591066:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829562.591070:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829562.591075:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829562.591087:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829562.591088:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564413696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829562.591101:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829562.591105:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829562.591108:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829562.591113:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4502 00000100:00080000:0.0:1787829562.591118:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829562.591123:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829562.591130:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829562.591146:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564413824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829562.591157:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829562.591161:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829562.591163:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829562.591167:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4502 00000100:00080000:0.0:1787829562.591172:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829562.591177:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829562.591183:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829562.591197:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564413952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829562.591211:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829562/real 1787829562] req@ffff976882aee940 x1874670564413440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829562.591344:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564413440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829562.591358:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829562.591365:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829562.591370:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564413440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829562.591457:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564413568:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829562.591483:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829562.591497:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829562/real 1787829562] req@ffff976882aec3c0 x1874670564413568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829562.591510:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564413568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829562.591519:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829562.591523:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829562.591526:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564413568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829562.591532:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564413696:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829562.591549:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829562.591559:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829562/real 1787829562] req@ffff976882aee580 x1874670564413696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829562.591579:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564413696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829562.591587:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829562.591591:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829562.591594:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564413696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829562.591600:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564413824:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829562.591620:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829562.591631:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829562/real 1787829562] req@ffff976882aed680 x1874670564413824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829562.591644:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564413824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829562.591651:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829562.591654:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829562.591657:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564413824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829562.591663:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564413952:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829562.591681:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829562.591691:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829562/real 1787829562] req@ffff976882aef0c0 x1874670564413952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829562.591702:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564413952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829562.591715:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829562.591719:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829562.591722:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564413952:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829562.846908:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829562.846908:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829562.846988:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829563.614927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829563.870930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829563.870933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829563.870955:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829564.638903:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829564.894896:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829564.894905:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829564.894945:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829565.662915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829565.918921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829565.918931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829565.918964:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829566.686903:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829566.942890:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829566.942902:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829566.942969:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829567.710957:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829567.710963:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829567.710973:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829567.710980:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829567.710993:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4507 00000100:00080000:0.0:1787829567.711007:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829567.711016:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829567.711067:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564414080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829567.711100:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829567.711104:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829567.711107:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829567.711112:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4507 00000100:00080000:0.0:1787829567.711117:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829567.711121:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829567.711133:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829567.711133:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564414080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829567.711156:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564414208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829567.711169:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829567.711174:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829567.711176:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829567.711185:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4507 00000100:00080000:0.0:1787829567.711190:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829567.711193:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829567.711197:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829567.711209:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564414336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829567.711223:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787829567.711223:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829567.711230:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829567.711233:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829567.711240:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4507 00000100:00080000:0.0:1787829567.711245:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829567.711249:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829567.711257:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829567.711269:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564414464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829567.711277:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829567.711280:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829567.711284:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829567.711285:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829567/real 1787829567] req@ffff976882aeed00 x1874670564414080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829567.711289:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4507 00000100:00080000:0.0:1787829567.711294:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829567.711300:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829567.711308:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829567.711319:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564414592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829567.711324:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564414080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829567.711350:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829567.711356:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829567.711361:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564414080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829567.711395:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564414208:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829567.711422:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829567.711436:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829567/real 1787829567] req@ffff976882aef0c0 x1874670564414208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829567.711463:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564414208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829567.711473:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829567.711476:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829567.711478:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564414208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829567.711483:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564414336:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829567.711499:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829567.711508:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829567/real 1787829567] req@ffff976882aed680 x1874670564414336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829567.711516:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564414336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829567.711522:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829567.711524:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829567.711526:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564414336:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829567.711529:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564414464:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829567.711540:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829567.711546:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829567/real 1787829567] req@ffff976882aee580 x1874670564414464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829567.711554:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564414464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829567.711564:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829567.711567:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829567.711569:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564414464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829567.711577:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564414592:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829567.711589:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829567.711595:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829567/real 1787829567] req@ffff976882aec3c0 x1874670564414592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829567.711603:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564414592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829567.711608:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829567.711611:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829567.711613:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564414592:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829567.966936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829567.966936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829567.967025:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829568.734970:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829568.990978:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829568.991001:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829568.991016:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829569.758915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829570.014968:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829570.014968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829570.015039:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829570.782907:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829571.038906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829571.038907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829571.038957:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829571.806919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829572.062924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829572.062924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829572.062992:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829572.830928:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829572.830933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829572.830942:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829572.830948:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829572.830958:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4512 00000100:00080000:3.0:1787829572.830969:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829572.830977:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829572.831017:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564414720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829572.831045:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829572.831049:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829572.831051:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829572.831055:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4512 00000100:00080000:3.0:1787829572.831059:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829572.831065:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829572.831075:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829572.831083:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564414720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829572.831092:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564414848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829572.831104:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829572.831108:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829572.831110:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829572.831114:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4512 00000100:00080000:3.0:1787829572.831119:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829572.831124:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829572.831129:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829572.831144:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564414976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829572.831154:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829572.831158:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829572.831161:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829572.831167:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4512 00000100:00080000:3.0:1787829572.831171:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829572.831176:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829572.831181:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564414848:192.168.203.139@tcp:38:kworker.0 00000008:00100000:3.0:1787829572.831184:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829572.831202:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564415104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829572.831211:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564414976:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787829572.831216:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829572.831222:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829572.831224:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829572.831230:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4512 00000100:00100000:1.0:1787829572.831234:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564415104:192.168.203.139@tcp:8:kworker.0 00000100:00080000:3.0:1787829572.831237:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829572.831245:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829572.831251:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829572.831268:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564415232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829572.831325:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564415232:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829573.086887:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829573.086899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829573.086963:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829573.854925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829574.110930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829574.110951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829574.110979:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829574.878940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829575.134894:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829575.134897:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829575.134971:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829575.902889:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:0.0:1787829575.903069:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:0.0:1787829575.903111:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829575.903129:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787829575.903168:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787829575.903178:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787829575.903190:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787829575.903203:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:1.0:1787829575.903218:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829572/real 1787829575] req@ffff97688cb5d2c0 x1874670564415104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829575.903244:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564415104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829575.903257:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829575.903262:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829575.903266:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564415104:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829575.903275:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829572/real 1787829575] req@ffff97688cb5c3c0 x1874670564414848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829575.903304:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564414848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829575.903313:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829575.903317:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829575.903321:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564414848:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829575.903328:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829572/real 1787829575] req@ffff97688cb5f0c0 x1874670564414976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829575.903342:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564414976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829575.903351:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829575.903359:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829575.903362:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564414976:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829575.903367:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829572/real 1787829575] req@ffff97688cb5c780 x1874670564414720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829575.903376:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564414720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829575.903385:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829575.903388:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829575.903392:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564414720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829575.903425:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829572/real 1787829575] req@ffff97688cb5e1c0 x1874670564415232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829575.903438:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564415232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829575.903447:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829575.903450:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829575.903454:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564415232:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829576.158905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829576.158907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829576.159010:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829576.926943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829577.182902:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829577.182908:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829577.182962:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829577.950892:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829577.950893:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829577.950901:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829577.950905:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829577.950913:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4517 00000100:00080000:3.0:1787829577.950921:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829577.950927:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829577.950956:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564415360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829577.950981:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829577.950986:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829577.950987:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829577.950990:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4517 00000100:00080000:3.0:1787829577.950993:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829577.950996:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829577.951003:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829577.951012:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564415488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829577.951021:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564415360:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787829577.951022:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829577.951030:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829577.951031:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829577.951035:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4517 00000100:00080000:3.0:1787829577.951037:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829577.951039:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829577.951043:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829577.951050:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564415616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829577.951056:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829577.951059:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829577.951060:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829577.951063:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4517 00000100:00080000:3.0:1787829577.951067:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829577.951070:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829577.951076:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829577.951089:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564415744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829577.951090:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787829577.951103:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829577.951107:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829577.951109:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829577.951114:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4517 00000100:00080000:3.0:1787829577.951118:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829577.951122:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829577.951128:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829577.951142:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564415872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829577.951148:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829577/real 1787829577] req@ffff97688cb5f840 x1874670564415360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829577.951172:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564415360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829577.951184:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829577.951189:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829577.951193:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564415360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829577.951219:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564415488:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829577.951245:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829577.951260:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829577/real 1787829577] req@ffff97688cb5d680 x1874670564415488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829577.951288:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564415488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829577.951299:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829577.951303:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829577.951307:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564415488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829577.951314:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564415616:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829577.951338:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829577.951350:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829577/real 1787829577] req@ffff97688cb5de00 x1874670564415616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829577.951362:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564415616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829577.951377:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829577.951380:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829577.951383:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564415616:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829577.951390:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564415744:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829577.951427:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829577.951441:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829577/real 1787829577] req@ffff97688cb5cb40 x1874670564415744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829577.951452:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564415744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829577.951459:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829577.951462:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829577.951465:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564415744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829577.951470:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564415872:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829577.951500:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829577.951510:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829577/real 1787829577] req@ffff97688cb5c000 x1874670564415872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829577.951520:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564415872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829577.951528:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829577.951531:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829577.951535:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564415872:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829578.206969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829578.206970:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829578.207028:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829578.974930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829579.230915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829579.230931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829579.230947:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829579.998904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829580.254970:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829580.254980:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829580.255043:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829581.022948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829581.278926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829581.278926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829581.278960:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829582.046894:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829582.302900:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829582.302900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829582.302997:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829583.070920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829583.070922:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829583.070930:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829583.070937:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829583.070946:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4522 00000100:00080000:1.0:1787829583.070956:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829583.070963:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829583.071003:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564416000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829583.071030:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829583.071033:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829583.071035:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829583.071039:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4522 00000100:00080000:1.0:1787829583.071043:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829583.071047:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829583.071058:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829583.071065:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564416000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829583.071073:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564416128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829583.071085:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829583.071088:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829583.071090:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829583.071094:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4522 00000100:00080000:1.0:1787829583.071099:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829583.071102:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829583.071107:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829583.071118:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564416256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829583.071127:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787829583.071135:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829583.071140:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829583.071142:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829583.071148:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4522 00000100:00080000:1.0:1787829583.071153:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829583.071157:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829583.071163:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829583.071166:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829583/real 1787829583] req@ffff976889eff0c0 x1874670564416000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829583.071176:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564416384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829583.071184:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564416000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829583.071186:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829583.071193:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829583.071196:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829583.071203:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4522 00000100:00080000:3.0:1787829583.071205:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829583.071206:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829583.071215:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:3.0:1787829583.071215:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000008:00100000:1.0:1787829583.071222:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829583.071224:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564416000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829583.071231:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564416512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829583.071262:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564416128:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829583.071281:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829583.071290:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829583/real 1787829583] req@ffff976889efe940 x1874670564416128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829583.071299:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564416128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829583.071305:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829583.071307:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829583.071310:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564416128:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829583.071313:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564416256:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829583.071324:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829583.071330:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829583/real 1787829583] req@ffff976889efc3c0 x1874670564416256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829583.071337:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564416256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829583.071342:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829583.071344:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829583.071346:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564416256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829583.071350:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564416384:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829583.071361:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829583.071367:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829583/real 1787829583] req@ffff976889effc00 x1874670564416384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829583.071375:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564416384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829583.071380:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829583.071382:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829583.071384:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564416384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829583.071387:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564416512:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829583.071397:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829583.071402:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829583/real 1787829583] req@ffff976889efed00 x1874670564416512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829583.071409:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564416512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829583.071414:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829583.071417:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829583.071420:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564416512:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829583.326932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829583.326932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829583.326964:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829584.094974:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829584.350912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829584.350933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829584.350969:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829585.118927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829585.374895:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829585.374905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829585.374915:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829586.142892:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829586.398900:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829586.398910:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829586.398992:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829587.166885:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829587.422931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829587.422932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829587.422964:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829588.190925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829588.190940:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829588.190948:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829588.190952:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829588.190962:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4528 00000100:00080000:2.0:1787829588.190974:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829588.190981:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829588.191028:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564416640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829588.191054:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829588.191058:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829588.191060:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829588.191064:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4528 00000100:00080000:2.0:1787829588.191069:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829588.191075:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829588.191084:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829588.191093:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564416640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829588.191100:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564416768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829588.191114:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829588.191117:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829588.191120:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829588.191124:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4528 00000100:00080000:2.0:1787829588.191139:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829588.191144:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829588.191149:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829588.191152:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829588.191167:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564416896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829588.191178:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829588.191181:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829588.191183:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829588.191190:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4528 00000100:00100000:3.0:1787829588.191190:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829588/real 1787829588] req@ffff976889cfa940 x1874670564416640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829588.191220:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829588.191228:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829588.191228:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564416640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829588.191238:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1787829588.191241:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829588.191246:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829588.191249:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564416640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829588.191255:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564417024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829588.191266:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829588.191270:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787829588.191271:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564416768:192.168.203.139@tcp:38:kworker.0 00000100:00080000:2.0:1787829588.191274:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829588.191281:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4528 00000100:00080000:2.0:1787829588.191286:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829588.191290:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:3.0:1787829588.191296:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:2.0:1787829588.191297:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829588.191309:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829588/real 1787829588] req@ffff976889cfb480 x1874670564416768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829588.191315:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564417152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829588.191323:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564416768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829588.191337:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829588.191339:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829588.191342:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564416768:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829588.191347:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564416896:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829588.191361:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829588.191368:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829588/real 1787829588] req@ffff976889cfad00 x1874670564416896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829588.191381:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564416896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829588.191389:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829588.191392:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829588.191396:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564416896:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829588.191401:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564417024:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829588.191423:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829588.191435:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829588/real 1787829588] req@ffff976889cfa1c0 x1874670564417024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829588.191446:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564417024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829588.191452:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829588.191454:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829588.191456:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564417024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829588.191468:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564417152:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829588.191482:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829588.191488:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829588/real 1787829588] req@ffff976889cf8f00 x1874670564417152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829588.191496:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564417152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829588.191505:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829588.191507:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829588.191509:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564417152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829588.446914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829588.446936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829588.446952:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829589.214920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829589.470928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829589.470944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829589.470945:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829590.238900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829590.494922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829590.494949:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829590.494992:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829591.262928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829591.518940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829591.518940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829591.518982:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829592.286876:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829592.542911:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829592.542911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829592.543290:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829593.310914:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829593.310919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829593.310922:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829593.310927:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829593.310934:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4533 00000100:00080000:3.0:1787829593.310943:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829593.310948:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829593.310979:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564417280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829593.311001:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829593.311004:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829593.311005:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829593.311008:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4533 00000100:00080000:3.0:1787829593.311010:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829593.311012:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829593.311019:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829593.311031:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564417408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829593.311042:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829593.311045:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564417280:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787829593.311047:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829593.311051:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829593.311057:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4533 00000100:00080000:3.0:1787829593.311061:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829593.311065:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829593.311071:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829593.311084:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564417536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829593.311094:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829593.311096:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787829593.311101:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829593.311104:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829593.311109:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4533 00000100:00080000:3.0:1787829593.311113:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829593.311117:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829593.311123:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829593.311128:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829593/real 1787829593] req@ffff97688cb5c000 x1874670564417280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829593.311137:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564417664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829593.311143:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564417280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829593.311161:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829593.311174:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829593.311178:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829593.311179:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:3.0:1787829593.311183:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829593.311184:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564417280:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787829593.311194:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4533 00000100:00080000:3.0:1787829593.311199:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829593.311203:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829593.311209:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829593.311211:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564417408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829593.311224:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564417792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829593.311247:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829593.311260:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829593/real 1787829593] req@ffff97688cb5cb40 x1874670564417408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829593.311268:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564417408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829593.311273:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829593.311276:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829593.311278:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564417408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829593.311281:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564417536:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829593.311300:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829593.311306:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829593/real 1787829593] req@ffff97688cb5de00 x1874670564417536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829593.311313:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564417536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829593.311318:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829593.311320:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829593.311322:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564417536:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829593.311325:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564417664:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829593.311334:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829593.311339:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829593/real 1787829593] req@ffff97688cb5d680 x1874670564417664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829593.311347:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564417664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829593.311355:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829593.311357:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829593.311360:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564417664:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829593.311378:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564417792:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829593.311396:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829593.311403:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829593/real 1787829593] req@ffff97688cb5f840 x1874670564417792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829593.311410:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564417792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829593.311415:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829593.311417:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829593.311419:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564417792:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829593.566909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829593.566909:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829593.566939:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829594.334870:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829594.590913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829594.590929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829594.590963:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829595.358924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829595.614923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829595.614943:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829595.615022:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829596.382934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829596.638906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829596.638907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829596.639027:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829597.406901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829597.662985:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829597.662986:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829597.663010:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829598.430937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829598.430971:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829598.430976:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829598.430979:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829598.430986:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4538 00000100:00080000:2.0:1787829598.430995:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829598.430999:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829598.431028:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564417920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829598.431049:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829598.431051:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829598.431053:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829598.431056:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4538 00000100:00080000:2.0:1787829598.431058:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829598.431061:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829598.431069:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829598.431083:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564417920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829598.431088:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564418048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829598.431096:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829598.431098:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829598.431100:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829598.431103:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4538 00000100:00080000:2.0:1787829598.431106:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829598.431108:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829598.431111:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829598.431119:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564418176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829598.431124:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829598.431140:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829598.431142:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829598.431148:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4538 00000800:00000100:3.0:1787829598.431149:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829598.431156:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829598.431159:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829598.431163:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829598.431174:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564418304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829598.431180:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829598.431182:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829598.431183:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829598.431186:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4538 00000100:00080000:2.0:1787829598.431189:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829598.431191:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829598.431198:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829598.431198:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829598/real 1787829598] req@ffff976889cf92c0 x1874670564417920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829598.431215:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564418432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829598.431225:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564417920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829598.431237:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829598.431240:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829598.431243:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564417920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829598.431264:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564418048:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829598.431330:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829598.431340:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829598/real 1787829598] req@ffff976889cf8f00 x1874670564418048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829598.431348:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564418048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829598.431354:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829598.431464:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829598.431468:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564418048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829598.431475:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564418176:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829598.431542:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829598.431573:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829598/real 1787829598] req@ffff976889cfa1c0 x1874670564418176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829598.431582:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564418176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829598.431587:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829598.431589:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829598.431591:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564418176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829598.431595:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564418304:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829598.431609:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829598.431615:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829598/real 1787829598] req@ffff976889cfad00 x1874670564418304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829598.431622:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564418304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829598.431626:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829598.431628:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829598.431631:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564418304:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829598.431634:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564418432:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829598.431643:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829598.431647:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829598/real 1787829598] req@ffff976889cfb480 x1874670564418432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829598.431659:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564418432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829598.431664:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829598.431665:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829598.431667:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564418432:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829598.686928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829598.686946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829598.687001:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829599.454897:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829599.710966:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829599.710966:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829599.710987:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829600.478942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829600.734937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829600.734992:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829600.735002:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829601.502868:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829601.758895:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829601.758903:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829601.758959:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829602.526972:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829602.782898:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829602.782954:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829602.782986:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829603.550968:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829603.550980:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829603.550982:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829603.550990:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829603.551004:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4543 00000100:00080000:0.0:1787829603.551019:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829603.551029:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829603.551078:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564418560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829603.551122:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829603.551126:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829603.551129:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829603.551142:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4543 00000100:00080000:0.0:1787829603.551148:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829603.551150:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564418560:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829603.551159:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829603.551174:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829603.551196:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564418688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829603.551214:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787829603.551214:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829603.551224:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829603.551227:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829603.551233:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4543 00000100:00080000:0.0:1787829603.551239:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829603.551244:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829603.551251:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829603.551254:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829603/real 1787829603] req@ffff976882aee940 x1874670564418560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829603.551273:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564418816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829603.551274:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564418560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829603.551319:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829603.551326:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:0.0:1787829603.551329:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829603.551331:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564418560:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829603.551345:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829603.551348:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829603.551355:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4543 00000100:00080000:0.0:1787829603.551362:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829603.551368:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829603.551373:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564418688:192.168.203.139@tcp:38:kworker.0 00000008:00100000:0.0:1787829603.551375:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829603.551398:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564418944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829603.551402:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829603.551420:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829603.551424:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829603/real 1787829603] req@ffff976882aec3c0 x1874670564418688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829603.551426:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829603.551430:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829603.551443:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4543 00000100:00080000:0.0:1787829603.551453:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829603.551458:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564418688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829603.551463:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:3.0:1787829603.551478:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829603.551480:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787829603.551483:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829603.551490:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564418688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829603.551495:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564418816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829603.551503:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564419072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829603.551516:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829603.551529:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829603/real 1787829603] req@ffff976882aee580 x1874670564418816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829603.551538:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564418816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829603.551543:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829603.551546:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829603.551552:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564418816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829603.551565:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564418944:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829603.551576:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829603.551582:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829603/real 1787829603] req@ffff976882aed680 x1874670564418944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829603.551589:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564418944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829603.551593:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829603.551595:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829603.551597:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564418944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829603.551600:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564419072:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829603.551609:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829603.551613:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829603/real 1787829603] req@ffff976882aef0c0 x1874670564419072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829603.551620:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564419072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829603.551624:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829603.551626:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829603.551627:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564419072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829603.806886:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829603.806897:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829603.806931:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829604.574929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829604.830920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:1.0:1787829604.830938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829604.830966:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829605.598955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829605.854903:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829605.854920:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829605.854935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829606.622933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829606.878951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829606.878974:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829606.879015:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829607.646929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829607.902899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829607.902913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829607.902963:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829608.670957:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829608.670962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829608.670967:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829608.670972:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829608.670980:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4548 00000100:00080000:1.0:1787829608.670989:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829608.670996:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829608.671029:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564419200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829608.671052:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829608.671054:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829608.671065:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829608.671068:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4548 00000100:00080000:1.0:1787829608.671070:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829608.671073:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829608.671081:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829608.671091:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564419328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829608.671099:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829608.671102:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829608.671105:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829608.671106:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564419200:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829608.671110:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4548 00000100:00080000:1.0:1787829608.671115:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829608.671118:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829608.671121:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829608.671130:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564419456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829608.671141:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829608.671144:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829608.671145:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829608.671148:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4548 00000100:00080000:1.0:1787829608.671151:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829608.671154:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829608.671157:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829608.671164:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564419584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829608.671171:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787829608.671175:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787829608.671176:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829608.671179:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829608.671183:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4548 00000100:00080000:1.0:1787829608.671186:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829608.671189:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829608.671192:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829608.671207:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564419712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829608.671219:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829608/real 1787829608] req@ffff976889efed00 x1874670564419200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829608.671249:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564419200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829608.671268:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829608.671272:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829608.671277:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564419200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829608.671301:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564419328:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829608.671322:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829608.671332:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829608/real 1787829608] req@ffff976889effc00 x1874670564419328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829608.671341:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564419328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829608.671347:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829608.671349:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829608.671351:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564419328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829608.671356:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564419456:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829608.671367:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829608.671373:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829608/real 1787829608] req@ffff976889efc3c0 x1874670564419456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829608.671381:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564419456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829608.671386:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829608.671388:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829608.671390:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564419456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829608.671394:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564419584:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829608.671408:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829608.671418:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829608/real 1787829608] req@ffff976889efe940 x1874670564419584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829608.671431:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564419584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829608.671439:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829608.671443:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829608.671446:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564419584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829608.671454:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564419712:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829608.671475:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829608.671487:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829608/real 1787829608] req@ffff976889eff0c0 x1874670564419712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829608.671496:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564419712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829608.671501:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829608.671503:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829608.671506:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564419712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829608.926937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829608.926937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829608.927005:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829609.694976:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829609.950931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829609.950956:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829609.950963:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829610.718963:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829610.974932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829610.974932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829610.974997:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829611.742904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829611.998912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829611.998917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829611.998953:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829612.766918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829613.022930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829613.022930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829613.022991:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829613.790900:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829613.790909:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1787829613.790910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829613.790913:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829613.790923:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4553 00000100:00080000:1.0:1787829613.790931:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829613.790937:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829613.790969:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564419840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829613.790989:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829613.790992:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829613.790993:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829613.790996:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4553 00000100:00080000:1.0:1787829613.790998:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829613.791000:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829613.791008:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829613.791016:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564419968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829613.791036:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829613.791037:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564419840:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829613.791039:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829613.791043:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829613.791047:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4553 00000100:00080000:1.0:1787829613.791050:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829613.791052:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829613.791056:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829613.791066:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564420096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829613.791074:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829613.791078:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829613.791080:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829613.791085:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4553 00000100:00080000:1.0:1787829613.791089:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829613.791093:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:3.0:1787829613.791098:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787829613.791099:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829613.791115:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564420224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829613.791126:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829613.791129:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829613.791130:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829613.791134:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4553 00000100:00080000:1.0:1787829613.791137:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829613.791142:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829613/real 1787829613] req@ffff976889efcb40 x1874670564419840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829613.791143:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829613.791149:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829613.791163:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564420352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829613.791173:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564419840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829613.791187:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829613.791192:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829613.791196:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564419840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829613.791221:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564419968:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829613.791248:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829613.791263:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829613/real 1787829613] req@ffff976889efe1c0 x1874670564419968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829613.791277:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564419968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829613.791286:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829613.791290:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829613.791293:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564419968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829613.791300:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564420096:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829613.791324:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829613.791337:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829613/real 1787829613] req@ffff976889efe580 x1874670564420096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829613.791350:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564420096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829613.791360:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829613.791363:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829613.791367:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564420096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829613.791374:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564420224:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829613.791398:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829613.791411:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829613/real 1787829613] req@ffff976889eff480 x1874670564420224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829613.791435:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564420224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829613.791442:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829613.791445:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829613.791447:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564420224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829613.791452:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564420352:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829613.791471:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829613.791482:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829613/real 1787829613] req@ffff976889efc780 x1874670564420352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829613.791495:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564420352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829613.791504:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829613.791508:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829613.791511:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564420352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829614.046914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829614.046914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829614.046934:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829614.814899:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829615.070957:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829615.070990:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829615.070998:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829615.838897:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829616.094883:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829616.094914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829616.094947:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829616.862888:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829617.118929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829617.118929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829617.118989:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829617.886950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829618.142880:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829618.142890:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829618.142909:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829618.910898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829618.910925:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829618.910930:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829618.910934:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829618.910942:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4558 00000100:00080000:0.0:1787829618.910950:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829618.910956:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829618.910998:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564420480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829618.911018:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829618.911020:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829618.911022:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829618.911025:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4558 00000100:00080000:0.0:1787829618.911032:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829618.911036:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829618.911045:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829618.911047:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564420480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829618.911059:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564420608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829618.911068:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829618.911070:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829618.911071:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829618.911074:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4558 00000100:00080000:0.0:1787829618.911077:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829618.911079:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829618.911084:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829618.911091:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564420736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829618.911098:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829618.911101:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829618.911102:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829618.911105:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4558 00000100:00080000:0.0:1787829618.911107:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829618.911111:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:3.0:1787829618.911114:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:0.0:1787829618.911122:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829618.911132:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564420864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829618.911139:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829618.911142:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829618.911143:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829618.911147:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4558 00000100:00080000:0.0:1787829618.911149:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829618.911153:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829618/real 1787829618] req@ffff976882aeed00 x1874670564420480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829618.911156:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829618.911164:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829618.911175:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564420480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829618.911177:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564420992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829618.911200:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829618.911209:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829618.911215:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564420480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829618.911236:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564420608:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829618.911259:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829618.911268:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829618/real 1787829618] req@ffff976882aef0c0 x1874670564420608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829618.911277:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564420608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829618.911283:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829618.911286:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829618.911288:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564420608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829618.911292:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564420736:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829618.911302:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829618.911308:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829618/real 1787829618] req@ffff976882aed680 x1874670564420736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829618.911315:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564420736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829618.911320:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829618.911322:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829618.911324:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564420736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829618.911327:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564420864:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829618.911336:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829618.911342:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829618/real 1787829618] req@ffff976882aee580 x1874670564420864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829618.911349:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564420864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829618.911359:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829618.911361:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829618.911363:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564420864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829618.911366:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564420992:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829618.911375:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829618.911380:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829618/real 1787829618] req@ffff976882aec3c0 x1874670564420992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829618.911387:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564420992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829618.911392:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829618.911393:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829618.911395:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564420992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829619.166908:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829619.166908:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829619.166948:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829619.934928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829620.190920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829620.190932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829620.190942:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829620.958884:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829621.214894:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829621.214896:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829621.214957:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829621.982936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829622.238890:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829622.238891:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829622.238928:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829623.006922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829623.262935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829623.262935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829623.262942:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829624.030927:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829624.030929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829624.030940:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829624.030946:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829624.030967:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4563 00000100:00080000:0.0:1787829624.030981:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829624.030990:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829624.031041:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564421120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829624.031071:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829624.031074:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829624.031077:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829624.031081:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4563 00000100:00080000:0.0:1787829624.031085:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829624.031090:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829624.031102:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829624.031103:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564421120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829624.031121:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564421248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829624.031133:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829624.031137:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829624.031140:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829624.031144:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4563 00000100:00080000:0.0:1787829624.031148:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829624.031153:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829624.031158:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829624.031171:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564421376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829624.031182:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787829624.031183:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829624.031190:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829624.031193:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829624.031199:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4563 00000100:00080000:0.0:1787829624.031204:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829624.031209:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829624.031214:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829624.031227:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564421504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829624.031232:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829624/real 1787829624] req@ffff976882aee940 x1874670564421120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829624.031244:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829624.031250:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829624.031253:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829624.031258:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564421120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829624.031261:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4563 00000100:00080000:0.0:1787829624.031267:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829624.031274:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:3.0:1787829624.031276:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829624.031279:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787829624.031282:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829624.031288:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564421120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829624.031296:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564421632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829624.031322:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564421248:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829624.031342:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829624.031352:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829624/real 1787829624] req@ffff976882aec3c0 x1874670564421248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829624.031361:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564421248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829624.031367:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829624.031370:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829624.031372:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564421248:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829624.031376:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564421376:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829624.031386:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829624.031398:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829624/real 1787829624] req@ffff976882aee580 x1874670564421376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829624.031407:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564421376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829624.031413:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829624.031415:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829624.031417:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564421376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829624.031422:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564421504:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829624.031434:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829624.031441:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829624/real 1787829624] req@ffff976882aed680 x1874670564421504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829624.031448:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564421504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829624.031454:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829624.031456:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829624.031458:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564421504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829624.031461:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564421632:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829624.031472:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829624.031477:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829624/real 1787829624] req@ffff976882aef0c0 x1874670564421632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829624.031484:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564421632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829624.031490:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829624.031491:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829624.031493:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564421632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829624.286941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829624.286958:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829624.286963:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829625.054922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829625.310903:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829625.310960:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829625.310986:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787829626.078914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829626.334975:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829626.334978:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829626.334998:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829627.102926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829627.358893:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829627.358976:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829627.359001:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829628.126952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829628.382989:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829628.382992:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829628.383060:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829629.150882:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829629.150904:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829629.150911:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829629.150927:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829629.150937:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4569 00000100:00080000:2.0:1787829629.150949:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829629.150956:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829629.150996:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564421760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829629.151021:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829629.151024:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829629.151026:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829629.151029:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4569 00000100:00080000:2.0:1787829629.151031:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829629.151035:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829629.151045:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829629.151051:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564421760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829629.151057:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564421888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829629.151070:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829629.151074:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829629.151076:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829629.151080:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4569 00000100:00080000:2.0:1787829629.151084:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829629.151088:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829629.151093:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829629.151107:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564422016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829629.151118:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829629.151122:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829629.151124:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829629.151128:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4569 00000100:00080000:2.0:1787829629.151131:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829629.151135:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:3.0:1787829629.151142:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:2.0:1787829629.151143:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829629.151155:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564422144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829629.151163:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829629.151166:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829629.151167:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829629.151170:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4569 00000100:00080000:2.0:1787829629.151174:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829629.151176:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829629.151179:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829629.151187:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564422272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829629.151198:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829629/real 1787829629] req@ffff976889cfa940 x1874670564421760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829629.151215:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564421760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829629.151226:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829629.151229:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829629.151232:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564421760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829629.151257:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564421888:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829629.151280:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829629.151315:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829629/real 1787829629] req@ffff976889cfbc00 x1874670564421888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829629.151329:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564421888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829629.151336:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829629.151339:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829629.151342:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564421888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829629.151346:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564422016:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829629.151365:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829629.151372:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829629/real 1787829629] req@ffff976889cf9e00 x1874670564422016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829629.151381:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564422016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829629.151387:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829629.151389:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829629.151391:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564422016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829629.151395:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564422144:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829629.151406:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829629.151412:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829629/real 1787829629] req@ffff976889cfa580 x1874670564422144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829629.151420:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564422144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829629.151425:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829629.151426:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829629.151428:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564422144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829629.151438:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564422272:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829629.151450:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829629.151457:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829629/real 1787829629] req@ffff976889cfb0c0 x1874670564422272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829629.151464:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564422272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829629.151475:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829629.151477:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829629.151479:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564422272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829629.406944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829629.406944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829629.407015:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829630.174956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829630.430930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829630.430945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829630.430957:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829631.198953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829631.454931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829631.454931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829631.455003:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829632.222982:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829632.478946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829632.478946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829632.479038:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829633.246883:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829633.502898:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829633.502899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829633.502997:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829634.270949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829634.270977:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829634.270984:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829634.270989:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829634.270999:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4574 00000100:00080000:2.0:1787829634.271011:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829634.271019:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829634.271060:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564422400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829634.271096:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829634.271101:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829634.271104:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829634.271109:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4574 00000100:00080000:2.0:1787829634.271126:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829634.271131:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829634.271142:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829634.271156:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564422528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829634.271166:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564422400:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829634.271176:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829634.271181:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829634.271184:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829634.271189:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4574 00000100:00080000:2.0:1787829634.271193:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829634.271198:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829634.271213:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829634.271230:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564422656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829634.271245:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829634.271250:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829634.271255:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829634.271276:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4574 00000100:00080000:2.0:1787829634.271285:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829634.271292:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829634.271301:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1787829634.271309:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829634.271320:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564422784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829634.271330:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829634.271356:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787829634.271385:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829634/real 1787829634] req@ffff976889cf8000 x1874670564422400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829634.271407:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564422400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829634.271418:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829634.271423:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:2.0:1787829634.271423:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829634.271434:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564422400:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829634.271440:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4574 00000100:00080000:2.0:1787829634.271467:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829634.271477:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829634.271484:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829634.271501:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564422528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829634.271510:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564422912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829634.271534:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829634.271556:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829634/real 1787829634] req@ffff976889cfb0c0 x1874670564422528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829634.271577:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564422528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829634.271585:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829634.271587:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829634.271590:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564422528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829634.271594:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564422656:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829634.271608:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829634.271617:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829634/real 1787829634] req@ffff976889cfa580 x1874670564422656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829634.271628:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564422656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829634.271634:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829634.271640:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829634.271643:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564422656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829634.271648:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564422784:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829634.271668:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829634.271675:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829634/real 1787829634] req@ffff976889cf9e00 x1874670564422784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829634.271697:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564422784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829634.271713:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829634.271719:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829634.271727:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564422784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829634.271767:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564422912:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829634.271787:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829634.271796:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829634/real 1787829634] req@ffff976889cfbc00 x1874670564422912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829634.271804:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564422912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829634.271809:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829634.271812:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829634.271814:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564422912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829634.526981:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829634.527011:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829634.527071:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829635.294968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829635.550928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829635.550936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829635.551048:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829636.318937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829636.574918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829636.574921:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829636.574988:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829637.342931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829637.598885:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829637.598908:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829637.598954:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829638.366906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829638.622908:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829638.622908:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829638.622972:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829639.390965:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829639.390983:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829639.390992:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829639.390995:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829639.391004:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4579 00000100:00080000:0.0:1787829639.391015:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829639.391020:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829639.391055:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564423040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829639.391084:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829639.391086:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829639.391088:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829639.391091:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4579 00000100:00080000:0.0:1787829639.391094:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829639.391098:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829639.391112:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829639.391122:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564423040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829639.391125:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564423168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829639.391140:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829639.391143:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829639.391144:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829639.391148:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4579 00000100:00080000:0.0:1787829639.391150:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829639.391153:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829639.391157:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829639.391165:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564423296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829639.391173:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829639.391175:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829639.391177:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829639.391180:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4579 00000100:00080000:0.0:1787829639.391186:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829639.391190:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829639.391194:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829639.391202:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564423424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829639.391209:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829639.391212:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829639.391212:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829639.391215:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4579 00000100:00080000:0.0:1787829639.391217:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829639.391220:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829639.391223:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829639.391237:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564423552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829639.391275:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564423168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829639.391315:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564423296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829639.391333:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564423424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829639.391348:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564423552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829639.646993:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829639.646993:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829639.647089:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829640.414915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829640.670933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829640.670954:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829640.671034:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829641.438942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829641.694930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829641.694930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829641.694999:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829642.462900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:2.0:1787829642.463020:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:2.0:1787829642.463062:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787829642.463076:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787829642.463111:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787829642.463121:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787829642.463128:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787829642.463137:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:3.0:1787829642.463192:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829639/real 1787829642] req@ffff976882aef0c0 x1874670564423168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829642.463226:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564423168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829642.463239:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829642.463244:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829642.463248:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564423168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829642.463256:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829639/real 1787829642] req@ffff976882aed680 x1874670564423296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829642.463268:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564423296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829642.463276:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829642.463279:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829642.463282:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564423296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829642.463288:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829639/real 1787829642] req@ffff976882aee580 x1874670564423424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829642.463300:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564423424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829642.463307:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829642.463311:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829642.463313:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564423424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829642.463320:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829639/real 1787829642] req@ffff976882aec3c0 x1874670564423552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829642.463332:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564423552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829642.463341:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829642.463344:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829642.463347:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564423552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829642.463353:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829639/real 1787829642] req@ffff976882aeed00 x1874670564423040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829642.463365:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564423040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829642.463373:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829642.463377:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829642.463380:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564423040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829642.718896:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829642.718906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829642.718967:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829643.486987:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829643.742927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829643.742935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829643.742993:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829644.510885:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829644.510893:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829644.510896:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829644.510904:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4584 00000100:00080000:0.0:1787829644.510913:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829644.510919:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829644.510950:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564423680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829644.510955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829644.510975:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829644.510978:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829644.510980:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829644.510983:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4584 00000100:00080000:0.0:1787829644.510985:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829644.510987:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829644.510994:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829644.511005:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564423808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829644.511012:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829644.511015:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829644.511016:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829644.511019:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4584 00000100:00080000:0.0:1787829644.511022:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829644.511024:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564423680:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829644.511027:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829644.511035:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829644.511046:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564423936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829644.511058:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829644.511061:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829644.511063:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829644.511069:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4584 00000100:00080000:0.0:1787829644.511073:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829644.511077:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829644.511082:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787829644.511093:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829644.511097:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564424064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829644.511110:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829644.511114:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829644.511116:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829644.511120:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4584 00000100:00080000:0.0:1787829644.511124:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829644.511129:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829644.511134:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829644.511138:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829644/real 1787829644] req@ffff976882aee940 x1874670564423680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829644.511149:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564424192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829644.511177:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564423680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829644.511187:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829644.511192:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829644.511196:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564423680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829644.511205:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564423808:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829644.511236:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829644.511250:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829644/real 1787829644] req@ffff976882aeed00 x1874670564423808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829644.511264:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564423808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829644.511273:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829644.511276:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829644.511279:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564423808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829644.511314:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564423936:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829644.511336:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829644.511348:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829644/real 1787829644] req@ffff976882aec3c0 x1874670564423936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829644.511360:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564423936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829644.511367:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829644.511370:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829644.511373:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564423936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829644.511378:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564424064:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829644.511395:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829644.511405:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829644/real 1787829644] req@ffff976882aee580 x1874670564424064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829644.511417:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564424064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829644.511426:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829644.511429:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829644.511432:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564424064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829644.511438:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564424192:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829644.511458:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829644.511468:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829644/real 1787829644] req@ffff976882aed680 x1874670564424192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829644.511481:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564424192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829644.511490:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829644.511493:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829644.511496:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564424192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829644.766886:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829644.766923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829644.766969:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829645.535072:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829645.790915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829645.790931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829645.790943:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829646.558910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829646.814951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829646.814951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829646.815029:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829647.582974:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829647.838905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829647.838911:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829647.838964:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829648.606924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829648.862912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829648.862914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829648.862981:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829649.630910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829649.630911:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829649.630920:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829649.630924:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829649.630931:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4589 00000100:00080000:0.0:1787829649.630940:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829649.630946:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829649.630996:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564424320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829649.631023:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829649.631028:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829649.631030:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829649.631051:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4589 00000100:00080000:0.0:1787829649.631056:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829649.631061:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829649.631069:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564424320:192.168.203.139@tcp:250:kworker.0 00000008:00100000:0.0:1787829649.631072:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829649.631086:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564424448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829649.631094:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829649.631097:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829649.631098:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829649.631101:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4589 00000100:00080000:0.0:1787829649.631104:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829649.631107:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829649.631112:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829649.631121:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564424576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829649.631127:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829649.631134:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829649.631137:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829649.631139:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829649.631143:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4589 00000100:00080000:0.0:1787829649.631148:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829649.631150:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829649.631153:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829649.631161:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564424704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829649.631167:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829649.631169:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829649.631171:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829649.631174:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4589 00000100:00080000:0.0:1787829649.631178:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829649.631178:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829649/real 1787829649] req@ffff976882aef0c0 x1874670564424320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829649.631185:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829649.631191:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829649.631200:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564424832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829649.631204:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564424320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829649.631219:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829649.631225:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829649.631230:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564424320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829649.631254:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564424448:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829649.631279:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829649.631290:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829649/real 1787829649] req@ffff976882aed680 x1874670564424448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829649.631299:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564424448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829649.631305:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829649.631308:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829649.631310:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564424448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829649.631314:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564424576:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829649.631327:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829649.631333:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829649/real 1787829649] req@ffff976882aee580 x1874670564424576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829649.631340:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564424576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829649.631346:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829649.631348:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829649.631351:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564424576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829649.631358:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564424704:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829649.631378:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829649.631390:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829649/real 1787829649] req@ffff976882aec3c0 x1874670564424704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829649.631403:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564424704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829649.631411:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829649.631413:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829649.631415:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564424704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829649.631420:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564424832:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829649.631435:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829649.631450:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829649/real 1787829649] req@ffff976882aeed00 x1874670564424832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829649.631458:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564424832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829649.631463:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829649.631466:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829649.631467:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564424832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829649.886930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829649.886930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829649.886996:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829650.654912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829650.910911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829650.910911:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829650.910943:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829651.678962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829651.934951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829651.934960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829651.935027:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829652.702962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829652.958934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829652.958935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829652.959046:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829653.726937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829653.982958:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829653.982958:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829653.983029:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829654.750941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829654.750962:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829654.750971:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829654.750976:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829654.750986:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4594 00000100:00080000:1.0:1787829654.750997:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829654.751004:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829654.751050:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564424960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829654.751079:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829654.751084:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829654.751086:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829654.751092:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4594 00000100:00080000:1.0:1787829654.751098:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829654.751103:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829654.751106:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564424960:192.168.203.139@tcp:250:kworker.0 00000008:00100000:1.0:1787829654.751115:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829654.751135:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564425088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829654.751147:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829654.751151:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829654.751155:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829654.751159:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4594 00000100:00080000:1.0:1787829654.751163:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829654.751166:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787829654.751171:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829654.751179:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829654.751195:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564425216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829654.751204:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829654/real 1787829654] req@ffff976889efc780 x1874670564424960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829654.751210:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829654.751218:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829654.751220:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829654.751225:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564424960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829654.751229:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4594 00000100:00080000:3.0:1787829654.751240:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829654.751244:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:1.0:1787829654.751248:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829654.751249:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564424960:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829654.751259:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829654.751266:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829654.751273:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564425088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829654.751284:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564425344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829654.751295:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787829654.751302:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829654.751309:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787829654.751309:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829654/real 1787829654] req@ffff976889eff480 x1874670564425088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829654.751313:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829654.751325:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4594 00000100:00100000:3.0:1787829654.751329:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564425088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829654.751331:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829654.751340:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:3.0:1787829654.751341:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829654.751348:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787829654.751356:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829654.751362:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564425088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829654.751367:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564425472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829654.751371:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564425216:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829654.751395:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829654.751404:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829654/real 1787829654] req@ffff976889efe580 x1874670564425216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829654.751413:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564425216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829654.751419:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829654.751421:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829654.751423:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564425216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829654.751434:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564425344:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829654.751445:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829654.751451:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829654/real 1787829654] req@ffff976889efe1c0 x1874670564425344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829654.751459:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564425344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829654.751464:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829654.751466:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829654.751468:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564425344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829654.751471:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564425472:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829654.751480:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829654.751485:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829654/real 1787829654] req@ffff976889efcb40 x1874670564425472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829654.751492:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564425472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829654.751501:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829654.751503:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829654.751505:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564425472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829655.006894:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829655.006897:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829655.006971:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829655.774896:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829656.030941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829656.030953:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829656.030982:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829656.798938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829657.054913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829657.054935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829657.055041:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829657.822882:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829658.078903:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829658.078912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829658.078997:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829658.846952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829659.102903:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829659.102903:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829659.102975:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829659.870888:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829659.870888:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829659.870897:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829659.870901:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829659.870908:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4599 00000100:00080000:1.0:1787829659.870917:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829659.870923:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829659.870954:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564425600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829659.870976:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829659.870979:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829659.870980:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829659.870983:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4599 00000100:00080000:1.0:1787829659.870986:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829659.870989:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829659.870997:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829659.871004:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564425600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829659.871009:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564425728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829659.871018:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829659.871021:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829659.871022:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829659.871025:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4599 00000100:00080000:1.0:1787829659.871027:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829659.871030:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829659.871033:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829659.871040:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564425856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829659.871046:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829659.871048:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829659.871050:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829659.871053:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4599 00000100:00080000:1.0:1787829659.871055:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829659.871058:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829659.871061:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787829659.871062:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829659.871073:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564425984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829659.871080:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829659.871083:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829659.871084:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829659.871086:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4599 00000100:00080000:1.0:1787829659.871089:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829659.871091:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829659.871095:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829659.871101:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564426112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829659.871110:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829659/real 1787829659] req@ffff976889eff0c0 x1874670564425600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829659.871124:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564425600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829659.871132:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829659.871135:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829659.871138:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564425600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829659.871159:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564425728:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829659.871181:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829659.871194:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829659/real 1787829659] req@ffff976889efe940 x1874670564425728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829659.871208:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564425728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829659.871217:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829659.871220:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829659.871230:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564425728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829659.871235:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564425856:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829659.871250:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829659.871258:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829659/real 1787829659] req@ffff976889efc3c0 x1874670564425856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829659.871266:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564425856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829659.871271:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829659.871273:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829659.871275:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564425856:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829659.871279:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564425984:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829659.871290:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829659.871295:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829659/real 1787829659] req@ffff976889effc00 x1874670564425984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829659.871303:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564425984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829659.871307:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829659.871309:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829659.871312:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564425984:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829659.871315:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564426112:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829659.871325:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829659.871337:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829659/real 1787829659] req@ffff976889efed00 x1874670564426112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829659.871344:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564426112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829659.871349:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829659.871351:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829659.871353:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564426112:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829660.126915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829660.126919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829660.126991:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829660.894986:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829661.150962:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829661.150962:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829661.151007:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829661.918927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829662.174887:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829662.174910:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829662.174990:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829662.942954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829663.198919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829663.198919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829663.198948:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829663.966922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829664.222906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829664.222907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829664.222944:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829664.990955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829664.990993:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829664.990997:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829664.990999:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829664.991006:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4604 00000100:00080000:3.0:1787829664.991019:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829664.991025:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829664.991060:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564426240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829664.991084:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829664.991086:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829664.991088:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829664.991090:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4604 00000100:00080000:3.0:1787829664.991093:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829664.991095:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829664.991103:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829664.991110:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564426368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829664.991117:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829664.991120:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829664.991121:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829664.991123:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4604 00000100:00080000:3.0:1787829664.991124:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829664.991127:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829664.991129:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829664.991134:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564426496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829664.991139:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829664.991142:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829664.991148:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829664.991151:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4604 00000100:00080000:3.0:1787829664.991153:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829664.991156:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829664.991160:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829664.991166:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564426624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829664.991173:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829664.991175:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829664.991176:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829664.991178:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4604 00000100:00080000:3.0:1787829664.991180:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829664.991183:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829664.991186:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829664.991193:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564426752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829664.991207:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564426240:192.168.203.139@tcp:250:kworker.0 00000800:00000100:0.0:1787829664.991321:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829664.991414:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829664/real 1787829664] req@ffff97688cb5e1c0 x1874670564426240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829664.991434:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564426240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829664.991445:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829664.991448:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829664.991451:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564426240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829664.991457:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564426368:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829664.991477:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829664.991494:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829664/real 1787829664] req@ffff97688cb5c780 x1874670564426368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829664.991503:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564426368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829664.991509:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829664.991512:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829664.991514:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564426368:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829664.991521:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564426496:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829664.991554:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829664.991563:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829664/real 1787829664] req@ffff97688cb5f0c0 x1874670564426496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829664.991573:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564426496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829664.991579:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829664.991582:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829664.991584:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564426496:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829664.991589:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564426624:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829664.991603:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829664.991611:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829664/real 1787829664] req@ffff97688cb5c3c0 x1874670564426624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829664.991620:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564426624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829664.991627:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829664.991629:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829664.991631:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564426624:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829664.991702:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564426752:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829664.991718:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829664.991726:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829664/real 1787829664] req@ffff97688cb5d2c0 x1874670564426752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829664.991735:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564426752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829664.991758:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829664.991761:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829664.991764:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564426752:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829665.246947:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829665.246973:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829665.246978:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829666.014902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829666.270908:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829666.270940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829666.270945:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829667.038898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829667.294945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829667.294945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829667.294977:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829668.062939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829668.318922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829668.318940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829668.319006:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829669.086920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829669.342933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829669.342933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829669.342955:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829670.110929:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829670.110940:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829670.110946:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829670.110958:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4610 00000100:00080000:3.0:1787829670.110972:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829670.110983:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829670.110993:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829670.111040:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564426880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829670.111092:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829670.111095:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829670.111097:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829670.111101:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4610 00000100:00080000:3.0:1787829670.111105:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829670.111109:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829670.111118:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829670.111132:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564427008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829670.111141:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829670.111145:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829670.111147:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829670.111152:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4610 00000100:00080000:3.0:1787829670.111156:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829670.111160:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829670.111165:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829670.111171:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564426880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829670.111179:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564427136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829670.111192:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829670.111207:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829670.111209:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829670.111213:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4610 00000100:00080000:3.0:1787829670.111217:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829670.111222:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829670.111227:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829670.111239:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564427264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829670.111248:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829670.111252:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787829670.111254:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829670.111257:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829670.111261:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4610 00000100:00080000:3.0:1787829670.111265:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829670.111268:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829670.111273:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829670.111283:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564427392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829670.111294:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829670/real 1787829670] req@ffff97688cb5d2c0 x1874670564426880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829670.111320:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564426880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829670.111329:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829670.111332:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829670.111336:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564426880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829670.111341:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564427008:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829670.111360:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829670.111370:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829670/real 1787829670] req@ffff97688cb5c3c0 x1874670564427008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829670.111381:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564427008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829670.111388:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829670.111391:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829670.111393:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564427008:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829670.111414:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564427136:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829670.111427:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829670.111441:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829670/real 1787829670] req@ffff97688cb5f0c0 x1874670564427136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829670.111448:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564427136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829670.111457:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829670.111460:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829670.111463:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564427136:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829670.111470:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564427264:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829670.111486:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829670.111503:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829670/real 1787829670] req@ffff97688cb5c780 x1874670564427264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829670.111510:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564427264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829670.111515:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829670.111518:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829670.111522:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564427264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829670.111527:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564427392:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829670.111540:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829670.111549:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829670/real 1787829670] req@ffff97688cb5e1c0 x1874670564427392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829670.111556:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564427392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829670.111569:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829670.111571:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829670.111573:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564427392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829670.366917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829670.366935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829670.366996:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829671.134912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829671.390952:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829671.390956:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829671.390989:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829672.158882:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829672.414949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829672.414951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829672.415030:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829673.182955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829673.438918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829673.438918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829673.438941:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829674.206923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829674.462879:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829674.462882:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829674.463032:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829675.230909:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829675.230917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829675.230917:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829675.230922:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829675.230930:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4615 00000100:00080000:3.0:1787829675.230939:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829675.230946:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829675.230978:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564427520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829675.230999:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829675.231002:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829675.231003:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829675.231006:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4615 00000100:00080000:3.0:1787829675.231008:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829675.231010:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829675.231018:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829675.231026:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564427648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829675.231035:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829675.231037:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829675.231039:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829675.231042:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4615 00000100:00100000:0.0:1787829675.231046:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564427520:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787829675.231046:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829675.231052:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829675.231057:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829675.231065:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564427776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829675.231071:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829675.231073:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829675.231075:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829675.231078:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4615 00000100:00080000:3.0:1787829675.231080:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829675.231083:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829675.231086:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829675.231093:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564427904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829675.231099:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829675.231102:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829675.231103:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829675.231107:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4615 00000100:00080000:3.0:1787829675.231111:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829675.231112:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787829675.231119:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829675.231126:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829675.231141:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564428032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829675.231152:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829675/real 1787829675] req@ffff97688cb5f840 x1874670564427520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829675.231168:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564427520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829675.231177:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829675.231180:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829675.231183:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564427520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829675.231206:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564427648:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829675.231221:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829675.231230:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829675/real 1787829675] req@ffff97688cb5d680 x1874670564427648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829675.231238:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564427648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829675.231244:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829675.231246:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829675.231249:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564427648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829675.231252:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564427776:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829675.231262:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829675.231268:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829675/real 1787829675] req@ffff97688cb5de00 x1874670564427776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829675.231275:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564427776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829675.231280:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829675.231282:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829675.231293:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564427776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829675.231296:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564427904:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829675.231309:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829675.231315:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829675/real 1787829675] req@ffff97688cb5cb40 x1874670564427904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829675.231323:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564427904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829675.231328:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829675.231330:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829675.231332:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564427904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829675.231335:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564428032:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829675.231345:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829675.231350:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829675/real 1787829675] req@ffff97688cb5c000 x1874670564428032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829675.231357:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564428032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829675.231362:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829675.231363:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829675.231365:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564428032:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829675.486927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829675.486927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829675.486948:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829676.254927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829676.510889:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829676.510895:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829676.510966:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829677.278898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829677.534911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829677.534911:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829677.534933:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829678.302917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829678.558922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829678.558925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829678.558983:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829679.326885:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829679.582914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829679.582917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829679.582945:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829680.350953:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829680.350955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829680.350962:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829680.350965:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829680.350974:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4620 00000100:00080000:2.0:1787829680.350983:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829680.350991:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829680.351027:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564428160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829680.351056:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829680.351058:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829680.351060:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829680.351063:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4620 00000100:00080000:2.0:1787829680.351066:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829680.351069:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829680.351076:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829680.351085:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564428288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829680.351092:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829680.351095:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829680.351096:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829680.351098:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4620 00000100:00080000:2.0:1787829680.351101:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829680.351103:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829680.351106:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829680.351112:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564428160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829680.351115:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564428416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829680.351125:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829680.351128:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829680.351129:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829680.351132:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4620 00000100:00080000:2.0:1787829680.351134:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829680.351137:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829680.351140:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829680.351148:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564428544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829680.351154:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829680.351157:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829680.351158:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829680.351161:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4620 00000100:00080000:2.0:1787829680.351164:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829680.351166:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829680.351170:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829680.351177:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564428672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829680.351186:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829680.351231:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829680/real 1787829680] req@ffff976889cfa940 x1874670564428160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829680.351248:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564428160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829680.351258:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829680.351262:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829680.351266:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564428160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829680.351273:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564428288:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829680.351298:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829680.351313:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829680/real 1787829680] req@ffff976889cfb480 x1874670564428288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829680.351325:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564428288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829680.351331:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829680.351334:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829680.351336:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564428288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829680.351362:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564428416:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829680.351378:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829680.351386:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829680/real 1787829680] req@ffff976889cfad00 x1874670564428416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829680.351394:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564428416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829680.351416:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829680.351419:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829680.351421:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564428416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829680.351426:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564428544:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829680.351438:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829680.351445:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829680/real 1787829680] req@ffff976889cfa1c0 x1874670564428544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829680.351453:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564428544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829680.351458:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829680.351460:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829680.351462:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564428544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829680.351474:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564428672:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829680.351498:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829680.351510:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829680/real 1787829680] req@ffff976889cf8f00 x1874670564428672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829680.351523:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564428672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829680.351530:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829680.351532:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829680.351534:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564428672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829680.606924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829680.606924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829680.607006:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829681.374894:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829681.630907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829681.630930:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829681.630934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787829682.398935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829682.654934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829682.654940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829682.654964:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829683.422921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829683.678913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829683.678913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829683.678951:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829684.446973:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829684.702930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829684.702930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829684.702998:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829685.470914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829685.470922:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829685.470927:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829685.470931:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829685.470939:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4625 00000100:00080000:1.0:1787829685.470949:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829685.470955:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829685.470988:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564428800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829685.471011:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829685.471014:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829685.471015:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829685.471018:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4625 00000100:00080000:1.0:1787829685.471020:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829685.471022:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829685.471029:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829685.471039:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564428928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829685.471047:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829685.471049:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829685.471052:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829685.471053:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564428800:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829685.471057:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4625 00000100:00080000:1.0:1787829685.471062:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829685.471065:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829685.471068:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829685.471075:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564429056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829685.471081:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829685.471084:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829685.471086:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829685.471100:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4625 00000100:00080000:1.0:1787829685.471103:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829685.471106:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829685.471109:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829685.471116:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564429184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829685.471122:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829685.471124:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829685.471125:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829685.471127:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4625 00000100:00080000:1.0:1787829685.471130:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829685.471135:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787829685.471135:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829685.471142:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829685.471150:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564429312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829685.471182:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829685/real 1787829685] req@ffff976889efed00 x1874670564428800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829685.471202:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564428800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829685.471210:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829685.471213:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829685.471216:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564428800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829685.471236:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564428928:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829685.471254:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829685.471265:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829685/real 1787829685] req@ffff976889effc00 x1874670564428928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829685.471273:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564428928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829685.471279:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829685.471281:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829685.471283:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564428928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829685.471287:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564429056:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829685.471298:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829685.471304:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829685/real 1787829685] req@ffff976889efc3c0 x1874670564429056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829685.471325:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564429056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829685.471331:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829685.471333:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829685.471335:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564429056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829685.471339:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564429184:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829685.471351:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829685.471358:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829685/real 1787829685] req@ffff976889efe940 x1874670564429184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829685.471370:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564429184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829685.471379:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829685.471382:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829685.471386:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564429184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829685.471393:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564429312:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829685.471414:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829685.471426:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829685/real 1787829685] req@ffff976889eff0c0 x1874670564429312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829685.471438:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564429312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829685.471449:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829685.471451:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829685.471454:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564429312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829685.726894:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829685.726898:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829685.726923:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829686.494906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829686.750946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829686.750947:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829686.750981:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829687.518984:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829687.774907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829687.774912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829687.774941:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829688.542907:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829688.798884:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829688.798887:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829688.798910:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829689.566966:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829689.822938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829689.822938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829689.822947:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829690.590908:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829690.590910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829690.590920:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829690.590925:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829690.590936:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4630 00000100:00080000:1.0:1787829690.590949:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829690.590957:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829690.590991:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564429440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829690.591016:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829690.591019:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829690.591021:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829690.591024:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4630 00000100:00080000:1.0:1787829690.591027:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829690.591030:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829690.591046:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829690.591056:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564429440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829690.591057:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564429568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829690.591070:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829690.591073:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829690.591074:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829690.591078:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4630 00000100:00080000:1.0:1787829690.591080:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829690.591082:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829690.591086:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829690.591095:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564429696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829690.591102:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829690.591105:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829690.591107:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829690.591110:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4630 00000100:00080000:1.0:1787829690.591113:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829690.591115:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:0.0:1787829690.591120:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787829690.591121:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829690.591132:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564429824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829690.591140:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829690.591142:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829690.591143:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829690.591146:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4630 00000100:00080000:1.0:1787829690.591148:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829690.591151:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829690.591155:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829690.591161:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829690/real 1787829690] req@ffff976889efcb40 x1874670564429440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829690.591165:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564429952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829690.591210:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564429440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829690.591219:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829690.591224:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829690.591234:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564429440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829690.591252:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564429568:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829690.591269:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829690.591278:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829690/real 1787829690] req@ffff976889efe1c0 x1874670564429568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829690.591286:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564429568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829690.591292:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829690.591294:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829690.591297:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564429568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829690.591301:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564429696:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829690.591312:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829690.591318:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829690/real 1787829690] req@ffff976889efe580 x1874670564429696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829690.591326:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564429696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829690.591331:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829690.591333:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829690.591334:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564429696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829690.591338:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564429824:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829690.591349:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829690.591355:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829690/real 1787829690] req@ffff976889eff480 x1874670564429824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829690.591362:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564429824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829690.591367:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829690.591369:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829690.591371:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564429824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829690.591374:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564429952:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829690.591384:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829690.591390:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829690/real 1787829690] req@ffff976889efc780 x1874670564429952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829690.591398:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564429952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829690.591403:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829690.591405:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829690.591406:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564429952:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829690.846919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829690.846919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829690.846929:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829691.614922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829691.870969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829691.870969:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829691.870978:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829692.638934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829692.894937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829692.894945:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829692.894993:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829693.662955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829693.918895:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829693.918895:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829693.918916:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829694.686947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829694.942960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829694.942960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829694.942997:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829695.710967:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829695.710967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829695.710982:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829695.710987:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829695.710998:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4635 00000100:00080000:2.0:1787829695.711011:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829695.711018:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829695.711068:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564430080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829695.711092:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829695.711095:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829695.711096:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829695.711099:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4635 00000100:00080000:2.0:1787829695.711102:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829695.711105:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829695.711116:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829695.711131:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564430208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829695.711139:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564430080:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829695.711142:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829695.711145:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829695.711149:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829695.711156:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4635 00000100:00080000:2.0:1787829695.711158:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829695.711161:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829695.711164:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829695.711173:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564430336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829695.711180:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829695.711183:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829695.711184:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829695.711187:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4635 00000100:00080000:2.0:1787829695.711190:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829695.711193:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829695.711197:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829695.711204:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564430464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829695.711210:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829695.711213:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829695.711214:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829695.711218:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4635 00000100:00080000:2.0:1787829695.711220:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829695.711225:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829695.711225:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829695.711233:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829695.711245:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564430592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829695.711282:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829695/real 1787829695] req@ffff976889cf92c0 x1874670564430080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829695.711304:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564430080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829695.711313:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829695.711317:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829695.711320:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564430080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829695.711345:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564430208:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829695.711365:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829695.711376:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829695/real 1787829695] req@ffff976889cf8f00 x1874670564430208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829695.711384:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564430208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829695.711395:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829695.711398:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829695.711400:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564430208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829695.711404:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564430336:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829695.711415:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829695.711421:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829695/real 1787829695] req@ffff976889cfa1c0 x1874670564430336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829695.711428:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564430336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829695.711434:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829695.711436:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829695.711438:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564430336:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829695.711442:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564430464:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829695.711470:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829695.711479:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829695/real 1787829695] req@ffff976889cfad00 x1874670564430464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829695.711487:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564430464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829695.711492:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829695.711495:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829695.711498:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564430464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829695.711502:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564430592:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829695.711514:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829695.711520:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829695/real 1787829695] req@ffff976889cfb480 x1874670564430592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829695.711527:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564430592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829695.711533:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829695.711535:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829695.711536:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564430592:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829695.966923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829695.966927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829695.966978:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829696.734943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829696.990961:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829696.990961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829696.991025:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829697.758939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829698.014949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829698.014951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829698.015083:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829698.782911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829699.038931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829699.038942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829699.038953:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829699.806924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829700.062972:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829700.062973:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829700.063024:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829700.830903:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829700.830914:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829700.830919:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829700.830923:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829700.830930:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4640 00000100:00080000:1.0:1787829700.830939:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829700.830945:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829700.830978:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564430720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829700.830999:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829700.831002:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829700.831003:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829700.831006:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4640 00000100:00080000:1.0:1787829700.831009:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829700.831011:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829700.831019:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829700.831030:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564430848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829700.831037:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564430720:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829700.831041:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829700.831044:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829700.831046:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829700.831049:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4640 00000100:00080000:1.0:1787829700.831051:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829700.831053:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829700.831057:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829700.831064:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564430976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829700.831071:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829700.831074:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829700.831075:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829700.831078:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4640 00000100:00080000:1.0:1787829700.831090:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829700.831093:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829700.831105:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1787829700.831114:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829700.831115:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564431104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829700.831123:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829700.831125:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829700.831126:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829700.831129:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4640 00000100:00080000:1.0:1787829700.831131:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829700.831133:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829700.831136:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829700.831141:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564431232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829700.831181:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829700/real 1787829700] req@ffff976889efc780 x1874670564430720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829700.831199:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564430720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829700.831208:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829700.831211:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829700.831214:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564430720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829700.831234:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564430848:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829700.831252:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829700.831261:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829700/real 1787829700] req@ffff976889eff480 x1874670564430848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829700.831270:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564430848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829700.831275:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829700.831278:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829700.831280:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564430848:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829700.831283:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564430976:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829700.831293:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829700.831299:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829700/real 1787829700] req@ffff976889efe580 x1874670564430976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829700.831306:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564430976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829700.831312:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829700.831313:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829700.831315:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564430976:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829700.831319:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564431104:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829700.831329:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829700.831335:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829700/real 1787829700] req@ffff976889efe1c0 x1874670564431104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829700.831343:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564431104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829700.831348:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829700.831350:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829700.831352:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564431104:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829700.831355:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564431232:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829700.831376:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829700.831383:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829700/real 1787829700] req@ffff976889efcb40 x1874670564431232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829700.831390:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564431232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829700.831395:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829700.831397:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829700.831399:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564431232:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829701.086903:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829701.086907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829701.086944:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829701.854925:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829702.110948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829702.110949:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829702.110993:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829702.878937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829703.134905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829703.134907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829703.134940:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829703.902939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829704.158988:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829704.158988:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829704.159048:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829704.926950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829705.182907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829705.182945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829705.182966:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829705.950918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829705.950951:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829705.950961:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829705.950967:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829705.950978:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4645 00000100:00080000:1.0:1787829705.950990:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829705.950999:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829705.951043:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564431360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829705.951072:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829705.951075:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829705.951078:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829705.951084:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4645 00000100:00080000:1.0:1787829705.951090:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829705.951101:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829705.951113:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829705.951122:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564431360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829705.951134:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564431488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829705.951147:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829705.951151:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829705.951153:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829705.951157:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4645 00000100:00080000:1.0:1787829705.951162:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829705.951166:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829705.951171:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829705.951183:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564431616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829705.951190:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829705.951193:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829705.951195:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829705.951203:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4645 00000100:00080000:1.0:1787829705.951207:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829705.951212:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829705.951218:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829705.951231:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564431744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829705.951242:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829705.951246:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829705.951248:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829705.951253:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4645 00000100:00080000:1.0:1787829705.951257:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829705.951264:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829705.951269:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829705.951282:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564431872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829705.951307:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564431488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829705.951471:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564431616:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829705.951509:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564431744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829705.951543:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564431872:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829706.206942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829706.206942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829706.207030:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829706.974935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829707.231007:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829707.231008:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829707.231053:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829707.999016:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829708.254953:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829708.254953:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829708.255020:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829709.022928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:0.0:1787829709.023077:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:0.0:1787829709.023108:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829709.023122:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787829709.023156:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787829709.023170:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787829709.023182:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787829709.023191:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:3.0:1787829709.023207:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829705/real 1787829709] req@ffff976889efe940 x1874670564431488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829709.023233:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564431488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829709.023245:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829709.023250:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829709.023254:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564431488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829709.023262:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829705/real 1787829709] req@ffff976889efc3c0 x1874670564431616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829709.023274:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564431616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829709.023283:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829709.023286:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829709.023290:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564431616:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829709.023297:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829705/real 1787829709] req@ffff976889effc00 x1874670564431744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829709.023308:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564431744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829709.023315:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829709.023319:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829709.023322:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564431744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829709.023328:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829705/real 1787829709] req@ffff976889efed00 x1874670564431872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829709.023340:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564431872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829709.023347:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829709.023351:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829709.023354:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564431872:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829709.023373:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829705/real 1787829709] req@ffff976889eff0c0 x1874670564431360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829709.023386:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564431360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829709.023395:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829709.023399:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829709.023402:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564431360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829709.278949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829709.278952:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829709.278993:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829710.046910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829710.302939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829710.302940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829710.303010:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829711.070931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829711.070945:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829711.070954:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829711.070959:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829711.070971:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4650 00000100:00080000:2.0:1787829711.070983:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829711.070990:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829711.071039:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564432000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829711.071067:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829711.071070:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829711.071072:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829711.071075:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4650 00000100:00080000:2.0:1787829711.071078:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829711.071080:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829711.071090:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829711.071100:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564432128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829711.071108:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829711.071111:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829711.071112:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829711.071116:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4650 00000100:00080000:2.0:1787829711.071120:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829711.071125:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564432000:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829711.071131:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829711.071140:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829711.071166:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564432256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829711.071180:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829711.071184:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829711.071187:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829711.071192:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4650 00000100:00080000:2.0:1787829711.071198:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829711.071201:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829711.071211:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829711.071219:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829711.071239:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564432384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829711.071250:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829711.071255:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829711/real 1787829711] req@ffff976889cfa940 x1874670564432000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829711.071257:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829711.071262:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829711.071281:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4650 00000100:00080000:2.0:1787829711.071292:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829711.071300:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829711.071308:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829711.071309:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564432000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829711.071330:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564432512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829711.071331:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829711.071363:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829711.071378:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564432000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829711.071401:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564432128:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829711.071445:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829711.071460:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829711/real 1787829711] req@ffff976889cfbc00 x1874670564432128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829711.071475:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564432128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829711.071483:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829711.071488:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829711.071491:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564432128:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829711.071523:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564432256:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829711.071546:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829711.071558:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829711/real 1787829711] req@ffff976889cf9e00 x1874670564432256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829711.071570:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564432256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829711.071578:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829711.071581:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829711.071584:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564432256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829711.071591:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564432384:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829711.071611:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829711.071623:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829711/real 1787829711] req@ffff976889cfa580 x1874670564432384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829711.071634:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564432384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829711.071642:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829711.071645:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829711.071648:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564432384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829711.071654:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564432512:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829711.071675:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829711.071685:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829711/real 1787829711] req@ffff976889cfb0c0 x1874670564432512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829711.071695:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564432512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829711.071702:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829711.071705:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829711.071708:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564432512:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829711.326911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829711.326969:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829711.326972:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829712.095118:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829712.350934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829712.350951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829712.350966:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829713.118938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829713.374909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829713.374933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829713.374971:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829714.142913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829714.398956:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829714.398956:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829714.398984:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829715.166942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829715.422951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829715.422955:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829715.422984:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829716.190917:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829716.190928:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829716.190934:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829716.190935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829716.190948:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4656 00000100:00080000:0.0:1787829716.190960:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829716.190968:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829716.191013:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564432640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829716.191038:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829716.191042:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829716.191044:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829716.191048:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4656 00000100:00080000:0.0:1787829716.191053:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829716.191059:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829716.191072:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829716.191079:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564432640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829716.191097:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564432768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829716.191113:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829716.191117:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829716.191120:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829716.191127:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4656 00000100:00080000:0.0:1787829716.191134:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829716.191139:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829716.191145:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829716.191150:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829716.191169:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564432896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829716.191180:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829716.191183:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829716.191185:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829716.191193:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4656 00000100:00080000:0.0:1787829716.191197:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829716.191201:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829716/real 1787829716] req@ffff976882aee940 x1874670564432640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829716.191206:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829716.191217:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829716.191228:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564432640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829716.191233:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564433024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829716.191285:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829716.191285:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829716.191300:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829716.191305:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829716.191305:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:0.0:1787829716.191333:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4656 00000100:00100000:3.0:1787829716.191333:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564432640:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829716.191342:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829716.191349:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829716.191355:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829716.191365:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564432768:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829716.191368:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564433152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829716.191400:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829716.191418:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829716/real 1787829716] req@ffff976882aeed00 x1874670564432768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829716.191427:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564432768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829716.191434:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829716.191436:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829716.191439:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564432768:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829716.191443:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564432896:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829716.191455:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829716.191461:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829716/real 1787829716] req@ffff976882aec3c0 x1874670564432896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829716.191468:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564432896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829716.191474:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829716.191476:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829716.191478:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564432896:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829716.191482:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564433024:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829716.191493:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829716.191499:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829716/real 1787829716] req@ffff976882aee580 x1874670564433024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829716.191506:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564433024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829716.191512:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829716.191514:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829716.191516:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564433024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829716.191529:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564433152:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829716.191541:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829716.191546:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829716/real 1787829716] req@ffff976882aed680 x1874670564433152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829716.191554:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564433152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829716.191559:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829716.191561:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829716.191563:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564433152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829716.446914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829716.446939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829716.446971:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829717.214915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829717.470944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829717.470950:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829717.470980:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829718.238914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829718.494923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829718.494955:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829718.494969:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829719.262938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829719.518980:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829719.518989:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829719.519017:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829720.286899:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829720.542907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829720.542933:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829720.543007:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00080000:0.0:1787829721.310986:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829721.310996:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829721.310998:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829721.311004:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829721.311016:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4661 00000100:00080000:0.0:1787829721.311028:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829721.311036:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829721.311079:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564433280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829721.311109:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829721.311112:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829721.311114:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829721.311117:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4661 00000100:00080000:0.0:1787829721.311121:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829721.311124:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829721.311133:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829721.311147:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564433408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829721.311158:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829721.311159:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564433280:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829721.311167:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829721.311175:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829721.311181:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4661 00000100:00080000:0.0:1787829721.311185:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829721.311189:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829721.311196:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829721.311211:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564433536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829721.311222:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829721.311226:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829721.311228:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829721.311233:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4661 00000100:00080000:0.0:1787829721.311237:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829721.311242:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829721.311250:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787829721.311267:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829721.311269:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564433664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829721.311302:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829721.311304:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829721.311306:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829721.311310:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4661 00000100:00080000:0.0:1787829721.311313:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829721.311316:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829721.311323:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829721.311336:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829721/real 1787829721] req@ffff976882aef0c0 x1874670564433280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829721.311337:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564433792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829721.311426:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564433280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829721.311439:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829721.311443:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829721.311447:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564433280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829721.311473:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564433408:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829721.311493:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829721.311503:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829721/real 1787829721] req@ffff976882aed680 x1874670564433408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829721.311512:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564433408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829721.311518:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829721.311521:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829721.311524:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564433408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829721.311535:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564433536:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829721.311548:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829721.311555:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829721/real 1787829721] req@ffff976882aee580 x1874670564433536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829721.311563:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564433536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829721.311569:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829721.311571:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829721.311574:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564433536:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829721.311577:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564433664:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829721.311595:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829721.311601:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829721/real 1787829721] req@ffff976882aec3c0 x1874670564433664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829721.311609:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564433664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829721.311615:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829721.311618:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829721.311620:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564433664:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829721.311623:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564433792:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829721.311633:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829721.311639:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829721/real 1787829721] req@ffff976882aeed00 x1874670564433792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829721.311651:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564433792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829721.311656:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829721.311658:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829721.311660:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564433792:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829721.566951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829721.566952:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829721.566981:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829722.334895:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829722.590932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829722.590937:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829722.591044:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787829723.358940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829723.614939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829723.614965:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829723.614975:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829724.382928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829724.638940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829724.638940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829724.638978:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829725.406938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829725.662930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829725.662956:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829725.662964:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829726.430906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829726.430957:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829726.430969:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829726.430974:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829726.430986:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4666 00000100:00080000:0.0:1787829726.430999:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829726.431008:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829726.431041:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564433920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829726.431065:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829726.431068:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829726.431069:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829726.431072:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4666 00000100:00080000:0.0:1787829726.431075:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829726.431078:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829726.431087:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829726.431097:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564434048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829726.431109:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564433920:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829726.431111:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829726.431125:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829726.431127:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829726.431131:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4666 00000100:00080000:0.0:1787829726.431133:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829726.431135:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829726.431139:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829726.431149:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564434176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829726.431156:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829726.431159:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829726.431161:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829726.431164:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4666 00000100:00080000:0.0:1787829726.431166:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829726.431169:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829726.431172:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829726.431179:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564434304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829726.431184:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829726.431188:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829726.431190:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829726.431192:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4666 00000100:00080000:0.0:1787829726.431195:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829726.431197:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829726.431201:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787829726.431208:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829726.431215:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564434432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829726.431279:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829726/real 1787829726] req@ffff976882aee940 x1874670564433920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829726.431301:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564433920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829726.431311:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829726.431314:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829726.431318:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564433920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829726.431347:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564434048:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829726.431368:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829726.431378:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829726/real 1787829726] req@ffff976882aeed00 x1874670564434048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829726.431386:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564434048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829726.431393:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829726.431397:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829726.431401:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564434048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829726.431407:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564434176:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829726.431428:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829726.431441:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829726/real 1787829726] req@ffff976882aec3c0 x1874670564434176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829726.431452:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564434176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829726.431458:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829726.431460:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829726.431463:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564434176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829726.431467:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564434304:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829726.431484:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829726.431494:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829726/real 1787829726] req@ffff976882aee580 x1874670564434304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829726.431507:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564434304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829726.431516:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829726.431520:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829726.431524:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564434304:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829726.431530:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564434432:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829726.431551:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829726.431562:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829726/real 1787829726] req@ffff976882aed680 x1874670564434432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829726.431575:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564434432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829726.431585:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829726.431589:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829726.431593:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564434432:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829726.686913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829726.686913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829726.686998:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829727.454932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829727.710965:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829727.711003:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829727.711006:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787829728.478946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829728.734917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829728.734918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829728.734994:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829729.502920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829729.758968:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829729.758975:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829729.758999:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829730.526931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829730.782920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829730.782923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829730.783025:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829731.550963:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829731.550975:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829731.550986:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829731.550993:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829731.551004:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4671 00000100:00080000:0.0:1787829731.551017:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829731.551024:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829731.551069:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564434560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829731.551097:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829731.551100:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829731.551102:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829731.551105:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4671 00000100:00080000:0.0:1787829731.551108:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829731.551110:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829731.551122:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829731.551135:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564434688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829731.551143:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829731.551150:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787829731.551151:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564434560:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829731.551152:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829731.551166:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4671 00000100:00080000:0.0:1787829731.551172:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829731.551180:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829731.551187:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829731.551219:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564434816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829731.551237:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829731.551241:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829731.551244:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829731.551249:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4671 00000100:00080000:0.0:1787829731.551257:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829731.551262:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829731.551267:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787829731.551270:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829731.551281:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564434944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829731.551290:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829731.551293:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829731.551295:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829731.551299:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4671 00000100:00080000:0.0:1787829731.551304:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829731.551309:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829731.551315:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829731.551322:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829731/real 1787829731] req@ffff976882aef0c0 x1874670564434560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829731.551334:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564435072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829731.551393:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564434560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829731.551407:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829731.551413:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829731.551418:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564434560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829731.551446:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564434688:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829731.551474:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829731.551489:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829731/real 1787829731] req@ffff976882aed680 x1874670564434688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829731.551503:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564434688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829731.551513:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829731.551517:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829731.551520:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564434688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829731.551527:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564434816:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829731.551547:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829731.551559:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829731/real 1787829731] req@ffff976882aee580 x1874670564434816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829731.551580:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564434816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829731.551590:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829731.551593:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829731.551605:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564434816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829731.551612:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564434944:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829731.551635:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829731.551648:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829731/real 1787829731] req@ffff976882aec3c0 x1874670564434944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829731.551662:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564434944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829731.551670:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829731.551674:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829731.551678:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564434944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829731.551692:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564435072:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829731.551714:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829731.551726:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829731/real 1787829731] req@ffff976882aeed00 x1874670564435072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829731.551739:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564435072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829731.551755:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829731.551758:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829731.551761:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564435072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829731.806924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829731.806949:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829731.806962:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829732.574942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829732.830915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829732.830962:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829732.831010:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829733.598902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829733.854925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829733.854938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829733.854977:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829734.622988:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829734.878909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829734.878924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829734.879005:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829735.646962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829735.902926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829735.902942:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829735.902968:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829736.670971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829736.670996:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829736.671005:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829736.671010:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829736.671020:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4676 00000100:00080000:2.0:1787829736.671034:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829736.671041:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829736.671087:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564435200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829736.671118:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829736.671121:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829736.671123:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829736.671126:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4676 00000100:00080000:2.0:1787829736.671128:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829736.671131:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829736.671141:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829736.671151:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564435328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829736.671161:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829736.671164:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829736.671181:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829736.671186:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564435200:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829736.671192:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4676 00000100:00080000:2.0:1787829736.671203:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829736.671209:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829736.671214:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829736.671227:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564435456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829736.671235:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829736.671238:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829736.671239:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829736.671245:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4676 00000100:00080000:2.0:1787829736.671257:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829736.671261:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829736.671266:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787829736.671268:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829736.671287:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564435584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829736.671297:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829736.671300:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829736.671303:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829736.671306:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4676 00000100:00080000:2.0:1787829736.671312:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829736.671318:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829736.671320:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829736/real 1787829736] req@ffff976889cf8000 x1874670564435200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829736.671323:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829736.671338:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564435712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829736.671348:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564435200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829736.671359:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829736.671364:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829736.671367:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564435200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829736.671387:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564435328:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829736.671404:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829736.671411:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829736/real 1787829736] req@ffff976889cfb0c0 x1874670564435328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829736.671419:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564435328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829736.671426:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829736.671428:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829736.671431:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564435328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829736.671434:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564435456:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829736.671445:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829736.671450:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829736/real 1787829736] req@ffff976889cfa580 x1874670564435456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829736.671458:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564435456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829736.671463:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829736.671465:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829736.671467:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564435456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829736.671470:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564435584:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829736.671481:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829736.671486:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829736/real 1787829736] req@ffff976889cf9e00 x1874670564435584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829736.671494:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564435584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829736.671499:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829736.671501:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829736.671503:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564435584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829736.671506:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564435712:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829736.671516:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829736.671521:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829736/real 1787829736] req@ffff976889cfbc00 x1874670564435712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829736.671528:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564435712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829736.671533:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829736.671535:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829736.671537:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564435712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829736.926887:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829736.926936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829736.927092:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829737.694946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829737.950918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829737.950952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829737.950964:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829738.718969:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829738.974933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829738.974973:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829738.975007:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829739.742932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829739.998939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829739.998943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829739.999009:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829740.766932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829741.022964:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829741.022964:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829741.022998:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829741.790918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829741.790923:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829741.790930:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829741.790933:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829741.790941:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4681 00000100:00080000:3.0:1787829741.790950:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829741.790956:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829741.790988:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564435840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829741.791013:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829741.791016:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829741.791018:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829741.791021:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4681 00000100:00080000:3.0:1787829741.791025:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829741.791029:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829741.791038:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829741.791059:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564435840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829741.791061:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564435968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829741.791079:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829741.791081:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829741.791083:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829741.791085:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4681 00000100:00080000:3.0:1787829741.791088:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829741.791092:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829741.791095:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829741.791107:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564436096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829741.791114:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829741.791116:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829741.791118:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829741.791121:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4681 00000100:00080000:3.0:1787829741.791124:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829741.791128:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:2.0:1787829741.791143:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:3.0:1787829741.791148:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829741.791160:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564436224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829741.791167:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829741.791170:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829741.791171:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829741.791185:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4681 00000100:00080000:3.0:1787829741.791189:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829741.791193:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829741/real 1787829741] req@ffff97688cb5c000 x1874670564435840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829741.791209:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564435840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829741.791216:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829741.791219:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829741.791222:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564435840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829741.791242:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564435968:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787829741.791248:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829741.791254:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829741.791262:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829741.791275:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829741/real 1787829741] req@ffff97688cb5cb40 x1874670564435968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829741.791283:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564435968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829741.791290:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829741.791292:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829741.791294:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564435968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829741.791298:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564436096:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829741.791311:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829741.791318:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829741/real 1787829741] req@ffff97688cb5de00 x1874670564436096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829741.791331:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564436096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829741.791337:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829741.791339:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829741.791342:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564436096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829741.791346:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564436224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829741.791355:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564436352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829741.791362:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829741.791382:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829741/real 1787829741] req@ffff97688cb5d680 x1874670564436224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829741.791394:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564436224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829741.791400:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829741.791402:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829741.791404:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564436224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829741.791428:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564436352:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829741.791442:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829741.791451:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829741/real 1787829741] req@ffff97688cb5f840 x1874670564436352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829741.791458:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564436352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829741.791467:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829741.791469:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829741.791471:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564436352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829742.046900:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829742.046991:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829742.046993:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829742.814906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829743.070938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829743.070954:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829743.070967:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829743.838916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829744.094916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829744.094916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829744.094969:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829744.862926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829745.118924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829745.118924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829745.118976:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829745.886947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829746.142934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829746.142934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829746.143001:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829746.910947:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829746.910952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829746.910956:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829746.910961:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829746.910969:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4686 00000100:00080000:0.0:1787829746.910978:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829746.910984:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829746.911016:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564436480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829746.911037:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829746.911040:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829746.911041:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829746.911045:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4686 00000100:00080000:0.0:1787829746.911047:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829746.911050:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829746.911058:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829746.911068:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564436608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829746.911075:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829746.911078:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829746.911079:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829746.911084:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4686 00000100:00100000:2.0:1787829746.911085:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564436480:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829746.911087:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829746.911105:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829746.911110:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829746.911118:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564436736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829746.911125:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829746.911127:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829746.911128:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829746.911131:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4686 00000100:00080000:0.0:1787829746.911134:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829746.911136:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829746.911140:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829746.911147:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564436864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829746.911154:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829746.911157:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829746.911158:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829746.911163:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829746.911166:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4686 00000100:00080000:0.0:1787829746.911170:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829746.911173:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829746.911177:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829746.911185:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564436992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829746.911207:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829746/real 1787829746] req@ffff976882aee940 x1874670564436480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829746.911223:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564436480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829746.911231:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829746.911234:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829746.911238:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564436480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829746.911256:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564436608:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829746.911271:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829746.911280:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829746/real 1787829746] req@ffff976882aeed00 x1874670564436608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829746.911287:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564436608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829746.911294:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829746.911296:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829746.911298:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564436608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829746.911301:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564436736:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829746.911311:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829746.911317:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829746/real 1787829746] req@ffff976882aec3c0 x1874670564436736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829746.911324:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564436736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829746.911330:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829746.911331:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829746.911333:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564436736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829746.911337:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564436864:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829746.911347:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829746.911353:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829746/real 1787829746] req@ffff976882aee580 x1874670564436864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829746.911359:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564436864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829746.911364:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829746.911366:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829746.911369:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564436864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829746.911372:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564436992:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829746.911381:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829746.911387:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829746/real 1787829746] req@ffff976882aed680 x1874670564436992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829746.911393:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564436992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829746.911398:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829746.911400:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829746.911402:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564436992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829747.166929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829747.166942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829747.166996:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829747.934884:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829748.190893:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829748.190943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829748.190954:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829748.958903:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829749.214909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829749.214921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829749.215011:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829749.982904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829750.238924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829750.238924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829750.238962:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829751.006915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829751.262943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829751.262949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829751.263075:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829752.030936:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829752.030942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829752.030945:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829752.030952:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829752.030960:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4691 00000100:00080000:0.0:1787829752.030969:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829752.030974:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829752.031019:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564437120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829752.031039:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829752.031042:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829752.031044:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829752.031047:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4691 00000100:00080000:0.0:1787829752.031049:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829752.031051:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829752.031071:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829752.031083:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564437248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829752.031090:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829752.031093:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829752.031095:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829752.031108:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4691 00000100:00080000:0.0:1787829752.031111:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829752.031114:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829752.031118:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829752.031124:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564437120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829752.031127:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564437376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829752.031138:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829752.031141:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829752.031143:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829752.031148:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4691 00000100:00080000:0.0:1787829752.031152:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829752.031156:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829752.031162:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829752.031176:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564437504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829752.031185:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829752.031193:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829752.031197:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829752.031199:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829752.031204:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4691 00000100:00080000:0.0:1787829752.031208:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829752.031212:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829752.031216:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829752.031226:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829752/real 1787829752] req@ffff976882aef0c0 x1874670564437120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829752.031229:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564437632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829752.031255:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564437120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829752.031265:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829752.031268:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829752.031273:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564437120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829752.031279:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564437248:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829752.031293:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829752.031301:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829752/real 1787829752] req@ffff976882aed680 x1874670564437248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829752.031309:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564437248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829752.031315:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829752.031328:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829752.031330:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564437248:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829752.031347:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564437376:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829752.031357:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829752.031363:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829752/real 1787829752] req@ffff976882aee580 x1874670564437376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829752.031371:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564437376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829752.031376:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829752.031379:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829752.031380:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564437376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829752.031384:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564437504:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829752.031393:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829752.031407:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829752/real 1787829752] req@ffff976882aec3c0 x1874670564437504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829752.031414:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564437504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829752.031419:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829752.031421:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829752.031423:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564437504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829752.031426:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564437632:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829752.031437:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829752.031443:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829752/real 1787829752] req@ffff976882aeed00 x1874670564437632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829752.031450:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564437632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829752.031455:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829752.031457:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829752.031458:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564437632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829752.286905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829752.286909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829752.286982:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829753.054900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829753.310914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829753.310923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829753.310944:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829754.078947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829754.334899:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829754.334907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829754.334959:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829755.102888:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829755.358900:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829755.358902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829755.358927:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829756.126946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829756.382894:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829756.382929:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829756.382977:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787829757.150878:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829757.150890:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829757.150895:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829757.150899:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829757.150907:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4697 00000100:00080000:0.0:1787829757.150916:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829757.150921:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829757.150954:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564437760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829757.150975:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829757.150978:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829757.150980:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829757.150982:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4697 00000100:00080000:0.0:1787829757.150985:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829757.150987:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829757.150994:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829757.151002:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564437888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829757.151009:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829757.151011:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829757.151012:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829757.151014:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4697 00000100:00080000:0.0:1787829757.151016:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829757.151018:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829757.151020:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829757.151039:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564438016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829757.151046:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829757.151048:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829757.151049:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829757.151052:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4697 00000100:00080000:0.0:1787829757.151055:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829757.151058:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829757.151063:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829757.151068:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564437760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829757.151073:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564438144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829757.151081:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829757.151083:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829757.151084:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829757.151086:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4697 00000100:00080000:0.0:1787829757.151088:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829757.151090:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829757.151093:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829757.151100:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564438272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829757.151133:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829757.151168:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829757/real 1787829757] req@ffff976882aee940 x1874670564437760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829757.151180:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564437760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829757.151188:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829757.151191:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829757.151194:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564437760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829757.151199:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564437888:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829757.151212:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829757.151219:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829757/real 1787829757] req@ffff976882aeed00 x1874670564437888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829757.151227:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564437888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829757.151233:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829757.151234:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829757.151236:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564437888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829757.151240:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564438016:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829757.151249:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829757.151255:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829757/real 1787829757] req@ffff976882aec3c0 x1874670564438016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829757.151262:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564438016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829757.151272:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829757.151274:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829757.151276:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564438016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829757.151295:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564438144:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829757.151306:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829757.151312:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829757/real 1787829757] req@ffff976882aee580 x1874670564438144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829757.151318:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564438144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829757.151324:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829757.151325:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829757.151328:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564438144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829757.151331:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564438272:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829757.151340:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829757.151346:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829757/real 1787829757] req@ffff976882aed680 x1874670564438272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829757.151353:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564438272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829757.151358:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829757.151359:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829757.151362:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564438272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829757.406900:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829757.406911:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829757.406941:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829758.175072:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829758.430907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829758.430928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829758.430940:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829759.198876:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829759.454895:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829759.454895:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829759.454926:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829760.222942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829760.478904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829760.478905:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829760.478942:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829761.246926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829761.502903:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829761.502910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829761.502940:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829762.270913:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829762.270916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829762.270924:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829762.270928:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829762.270936:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4702 00000100:00080000:0.0:1787829762.270945:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829762.270952:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829762.270983:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564438400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829762.271024:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829762.271027:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829762.271028:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829762.271032:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4702 00000100:00080000:0.0:1787829762.271034:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829762.271037:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829762.271045:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829762.271057:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564438528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829762.271064:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829762.271066:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829762.271068:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829762.271070:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4702 00000100:00080000:0.0:1787829762.271072:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829762.271074:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829762.271076:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829762.271087:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564438656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829762.271087:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564438400:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829762.271101:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829762.271103:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829762.271104:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829762.271107:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4702 00000100:00080000:0.0:1787829762.271110:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829762.271112:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829762.271115:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829762.271123:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564438784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829762.271130:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829762.271132:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829762.271133:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829762.271135:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4702 00000100:00080000:0.0:1787829762.271138:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829762.271140:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829762.271143:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829762.271150:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564438912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829762.271153:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829762.271197:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829762/real 1787829762] req@ffff976882aef0c0 x1874670564438400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829762.271212:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564438400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829762.271220:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829762.271223:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829762.271226:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564438400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829762.271231:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564438528:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829762.271245:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829762.271253:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829762/real 1787829762] req@ffff976882aed680 x1874670564438528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829762.271260:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564438528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829762.271266:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829762.271269:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829762.271271:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564438528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829762.271290:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564438656:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829762.271300:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829762.271306:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829762/real 1787829762] req@ffff976882aee580 x1874670564438656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829762.271314:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564438656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829762.271318:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829762.271320:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829762.271322:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564438656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829762.271325:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564438784:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829762.271335:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829762.271341:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829762/real 1787829762] req@ffff976882aec3c0 x1874670564438784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829762.271348:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564438784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829762.271353:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829762.271354:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829762.271356:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564438784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829762.271359:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564438912:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829762.271368:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829762.271374:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829762/real 1787829762] req@ffff976882aeed00 x1874670564438912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829762.271381:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564438912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829762.271386:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829762.271387:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829762.271389:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564438912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829762.526927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829762.526928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829762.527027:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829763.294898:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829763.550907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829763.550909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829763.550930:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829764.318964:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829764.574922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829764.574922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829764.574993:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829765.342921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829765.598946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829765.598991:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829765.599039:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829766.367004:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829766.622947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829766.622950:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829766.623041:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829767.390979:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829767.390990:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829767.390995:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829767.390999:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829767.391014:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4707 00000100:00080000:0.0:1787829767.391027:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829767.391036:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829767.391075:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564439040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829767.391102:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829767.391106:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829767.391108:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829767.391112:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4707 00000100:00080000:0.0:1787829767.391116:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829767.391120:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829767.391130:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829767.391145:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564439168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829767.391157:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829767.391161:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829767.391165:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829767.391166:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564439040:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829767.391179:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4707 00000100:00080000:0.0:1787829767.391183:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829767.391187:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829767.391192:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829767.391206:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564439296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829767.391216:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829767.391219:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829767.391222:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829767.391226:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4707 00000100:00080000:0.0:1787829767.391230:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829767.391237:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:2.0:1787829767.391237:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:0.0:1787829767.391245:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829767.391259:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564439424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829767.391271:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829767.391274:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829767.391276:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829767.391281:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4707 00000100:00080000:0.0:1787829767.391285:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829767.391287:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829767/real 1787829767] req@ffff976882aee940 x1874670564439040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829767.391292:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829767.391299:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829767.391315:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564439552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829767.391320:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564439040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829767.391341:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829767.391346:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829767.391360:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564439040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829767.391383:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564439168:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829767.391402:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829767.391416:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829767/real 1787829767] req@ffff976882aeed00 x1874670564439168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829767.391429:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564439168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829767.391439:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829767.391442:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829767.391446:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564439168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829767.391453:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564439296:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829767.391473:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829767.391482:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829767/real 1787829767] req@ffff976882aec3c0 x1874670564439296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829767.391490:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564439296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829767.391496:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829767.391498:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829767.391500:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564439296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829767.391504:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564439424:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829767.391516:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829767.391522:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829767/real 1787829767] req@ffff976882aee580 x1874670564439424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829767.391529:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564439424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829767.391535:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829767.391537:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829767.391540:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564439424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829767.391543:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564439552:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829767.391570:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829767.391578:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829767/real 1787829767] req@ffff976882aed680 x1874670564439552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829767.391590:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564439552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829767.391599:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829767.391602:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829767.391606:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564439552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829767.646923:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829767.646923:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829767.647009:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829768.414921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829768.670922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829768.670922:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829768.670963:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829769.438923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829769.694932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829769.694981:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829769.695013:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829770.462996:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829770.718921:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829770.718922:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829770.718999:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829771.486982:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829771.742907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829771.742944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829771.743038:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829772.510964:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829772.510980:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829772.510988:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829772.510994:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829772.511008:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4712 00000100:00080000:3.0:1787829772.511023:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829772.511033:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829772.511083:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564439680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829772.511114:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829772.511118:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829772.511121:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829772.511125:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4712 00000100:00080000:3.0:1787829772.511130:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829772.511134:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829772.511147:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829772.511163:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564439808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829772.511175:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564439680:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787829772.511195:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829772.511201:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829772.511203:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829772.511210:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4712 00000100:00080000:3.0:1787829772.511216:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829772.511222:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829772.511229:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829772.511246:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564439936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829772.511258:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829772.511262:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829772.511264:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829772.511269:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4712 00000100:00080000:3.0:1787829772.511273:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829772.511278:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829772.511285:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829772.511304:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564440064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829772.511319:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564439808:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787829772.511332:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829772.511339:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829772.511342:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829772.511348:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4712 00000100:00080000:3.0:1787829772.511354:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829772.511359:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829772.511366:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829772.511372:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564439936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829772.511389:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564440192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829772.511420:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564440064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829772.511458:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564440192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829772.766980:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829772.766985:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829772.767057:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829773.535126:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829773.790891:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829773.790959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829773.791006:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829774.558994:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829774.814919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829774.814928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829774.814972:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829775.583052:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:2.0:1787829775.583150:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:2.0:1787829775.583201:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787829775.583218:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787829775.583258:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787829775.583267:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787829775.583275:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:2.0:1787829775.583281:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:3.0:1787829775.583340:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829772/real 1787829775] req@ffff97688cb5c3c0 x1874670564440064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829775.583370:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564440064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829775.583382:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829775.583387:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829775.583392:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564440064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829775.583400:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829772/real 1787829775] req@ffff97688cb5d2c0 x1874670564440192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829775.583412:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564440192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829775.583419:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829775.583423:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829775.583426:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564440192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829775.583432:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829772/real 1787829775] req@ffff97688cb5c780 x1874670564439808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829775.583444:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564439808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829775.583452:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829775.583456:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829775.583459:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564439808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829775.583465:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829772/real 1787829775] req@ffff97688cb5f0c0 x1874670564439936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829775.583476:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564439936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829775.583484:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829775.583487:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829775.583491:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564439936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829775.583496:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829772/real 1787829775] req@ffff97688cb5e1c0 x1874670564439680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829775.583507:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564439680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829775.583514:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829775.583518:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829775.583521:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564439680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829775.838910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829775.838910:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829775.838963:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829776.606961:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829776.862927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829776.862927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829776.862979:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829777.630921:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829777.630936:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829777.630940:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829777.630949:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4717 00000100:00080000:3.0:1787829777.630960:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829777.630967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829777.630967:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829777.630998:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564440320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829777.631028:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829777.631033:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829777.631036:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829777.631040:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4717 00000100:00080000:3.0:1787829777.631044:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829777.631049:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829777.631060:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829777.631073:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564440448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829777.631084:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564440320:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787829777.631084:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829777.631088:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829777.631091:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829777.631099:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4717 00000100:00080000:3.0:1787829777.631103:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829777.631106:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829777.631111:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829777.631123:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564440576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829777.631134:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829777.631137:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829777.631139:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829777.631145:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4717 00000100:00080000:3.0:1787829777.631148:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829777.631155:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:2.0:1787829777.631157:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:3.0:1787829777.631161:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829777.631174:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564440704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829777.631182:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829777.631185:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829777.631186:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829777.631189:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4717 00000100:00080000:3.0:1787829777.631192:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829777.631199:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829777.631200:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829777/real 1787829777] req@ffff97688cb5e1c0 x1874670564440320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829777.631204:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829777.631221:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564440832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829777.631227:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564440320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829777.631240:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829777.631245:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829777.631249:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564440320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829777.631275:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564440448:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829777.631297:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829777.631309:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829777/real 1787829777] req@ffff97688cb5f0c0 x1874670564440448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829777.631321:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564440448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829777.631329:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829777.631332:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829777.631335:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564440448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829777.631340:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564440576:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829777.631369:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829777.631380:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829777/real 1787829777] req@ffff97688cb5c780 x1874670564440576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829777.631389:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564440576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829777.631396:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829777.631399:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829777.631402:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564440576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829777.631407:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564440704:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829777.631425:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829777.631434:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829777/real 1787829777] req@ffff97688cb5d2c0 x1874670564440704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829777.631443:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564440704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829777.631450:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829777.631453:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829777.631456:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564440704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829777.631461:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564440832:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829777.631476:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829777.631486:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829777/real 1787829777] req@ffff97688cb5c3c0 x1874670564440832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829777.631496:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564440832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829777.631503:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829777.631506:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829777.631509:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564440832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829777.886929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829777.886929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829777.887000:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829778.655019:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829778.910953:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829778.910962:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829778.910995:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829779.678952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829779.934939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829779.934939:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829779.934995:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829780.702972:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829780.958902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829780.958944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829780.958963:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829781.726990:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829781.982948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829781.982965:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829781.983010:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829782.750953:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829782.750954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829782.750964:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829782.750970:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829782.750982:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4722 00000100:00080000:3.0:1787829782.750993:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829782.751001:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829782.751049:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564440960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829782.751083:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829782.751087:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829782.751089:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829782.751093:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4722 00000100:00080000:3.0:1787829782.751098:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829782.751102:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829782.751114:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829782.751127:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564440960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829782.751130:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564441088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829782.751144:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829782.751147:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829782.751149:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829782.751153:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4722 00000100:00080000:3.0:1787829782.751157:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829782.751161:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829782.751166:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829782.751178:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564441216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829782.751187:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829782.751190:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829782.751192:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829782.751195:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4722 00000100:00080000:3.0:1787829782.751199:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829782.751204:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787829782.751205:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829782.751212:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829782.751223:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564441344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829782.751237:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829782.751241:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829782.751243:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829782.751247:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4722 00000100:00080000:3.0:1787829782.751251:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829782.751256:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829782/real 1787829782] req@ffff97688cb5c3c0 x1874670564440960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829782.751257:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829782.751265:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829782.751276:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564440960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829782.751280:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564441472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829782.751292:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829782.751297:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829782.751302:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564440960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829782.751324:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564441088:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829782.751347:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829782.751360:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829782/real 1787829782] req@ffff97688cb5d2c0 x1874670564441088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829782.751371:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564441088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829782.751376:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829782.751378:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829782.751380:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564441088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829782.751384:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564441216:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829782.751407:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829782.751414:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829782/real 1787829782] req@ffff97688cb5c780 x1874670564441216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829782.751421:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564441216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829782.751426:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829782.751428:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829782.751430:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564441216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829782.751434:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564441344:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829782.751445:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829782.751450:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829782/real 1787829782] req@ffff97688cb5f0c0 x1874670564441344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829782.751456:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564441344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829782.751461:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829782.751462:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829782.751464:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564441344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829782.751472:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564441472:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829782.751482:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829782.751487:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829782/real 1787829782] req@ffff97688cb5e1c0 x1874670564441472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829782.751493:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564441472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829782.751498:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829782.751499:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829782.751501:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564441472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829783.006950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829783.006950:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829783.007008:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829783.774948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829784.030946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829784.030967:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829784.030975:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829784.798880:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829785.054945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829785.054953:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829785.054995:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829785.822900:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829786.078934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829786.078947:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829786.078959:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829786.846939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829787.102982:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829787.102982:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829787.103059:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829787.870943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829787.870953:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829787.870959:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829787.870962:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829787.870970:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4727 00000100:00080000:0.0:1787829787.870980:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829787.870986:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829787.871021:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564441600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829787.871044:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829787.871046:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829787.871047:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829787.871050:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4727 00000100:00080000:0.0:1787829787.871052:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829787.871055:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829787.871062:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829787.871070:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564441728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829787.871077:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829787.871080:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829787.871081:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829787.871084:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4727 00000100:00080000:0.0:1787829787.871087:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829787.871089:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829787.871093:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829787.871100:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564441856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829787.871106:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829787.871109:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829787.871110:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829787.871113:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4727 00000100:00080000:0.0:1787829787.871115:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829787.871119:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829787.871122:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829787.871130:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564441984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829787.871142:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829787.871143:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564441600:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829787.871161:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829787.871164:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829787.871168:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4727 00000100:00080000:0.0:1787829787.871170:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829787.871173:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829787.871180:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829787.871211:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564442112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829787.871265:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829787.871328:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829787/real 1787829787] req@ffff976882aef0c0 x1874670564441600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829787.871345:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564441600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829787.871356:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829787.871361:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829787.871365:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564441600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829787.871374:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564441728:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829787.871400:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829787.871414:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829787/real 1787829787] req@ffff976882aed680 x1874670564441728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829787.871423:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564441728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829787.871429:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829787.871432:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829787.871434:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564441728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829787.871439:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564441856:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829787.871454:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829787.871460:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829787/real 1787829787] req@ffff976882aee580 x1874670564441856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829787.871468:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564441856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829787.871473:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829787.871475:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829787.871477:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564441856:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829787.871504:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564441984:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829787.871515:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829787.871521:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829787/real 1787829787] req@ffff976882aec3c0 x1874670564441984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829787.871529:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564441984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829787.871534:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829787.871536:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829787.871538:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564441984:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829787.871542:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564442112:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829787.871551:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829787.871557:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829787/real 1787829787] req@ffff976882aeed00 x1874670564442112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829787.871564:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564442112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829787.871573:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829787.871575:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829787.871578:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564442112:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829788.126983:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829788.126995:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829788.127009:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829788.894914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829789.150929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829789.150953:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829789.150967:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829789.918915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829790.174921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829790.174966:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829790.174972:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829790.942881:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829791.198931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829791.198970:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829791.199001:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829791.966934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829792.222976:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829792.222976:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829792.222976:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829792.990943:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829792.990944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829792.990953:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829792.990958:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829792.990969:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4732 00000100:00080000:1.0:1787829792.990979:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829792.990986:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829792.991025:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564442240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829792.991050:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829792.991054:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829792.991056:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829792.991060:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4732 00000100:00080000:1.0:1787829792.991064:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829792.991068:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829792.991078:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829792.991092:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564442368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829792.991102:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564442240:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829792.991120:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829792.991124:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829792.991126:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829792.991131:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4732 00000100:00080000:1.0:1787829792.991136:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829792.991140:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829792.991145:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829792.991159:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564442496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829792.991171:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829792.991171:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787829792.991178:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829792.991181:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829792.991187:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4732 00000100:00080000:1.0:1787829792.991191:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829792.991196:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829792.991201:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829792.991213:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564442624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829792.991221:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829792/real 1787829792] req@ffff976889eff0c0 x1874670564442240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829792.991237:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829792.991242:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829792.991244:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564442240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829792.991245:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829792.991254:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4732 00000100:00080000:1.0:1787829792.991263:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829792.991266:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829792.991269:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:2.0:1787829792.991274:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000008:00100000:1.0:1787829792.991276:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829792.991283:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564442240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829792.991293:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564442752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829792.991313:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564442368:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829792.991335:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829792.991345:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829792/real 1787829792] req@ffff976889efed00 x1874670564442368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829792.991354:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564442368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829792.991360:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829792.991362:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829792.991365:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564442368:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829792.991369:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564442496:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829792.991379:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829792.991385:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829792/real 1787829792] req@ffff976889effc00 x1874670564442496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829792.991392:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564442496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829792.991397:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829792.991399:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829792.991401:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564442496:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829792.991404:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564442624:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829792.991415:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829792.991421:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829792/real 1787829792] req@ffff976889efc3c0 x1874670564442624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829792.991428:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564442624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829792.991433:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829792.991435:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829792.991437:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564442624:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829792.991440:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564442752:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829792.991449:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829792.991454:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829792/real 1787829792] req@ffff976889efe940 x1874670564442752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829792.991461:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564442752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829792.991465:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829792.991467:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829792.991469:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564442752:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829793.246899:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829793.246899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829793.246970:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829794.014927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829794.270926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829794.270927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829794.270957:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829795.038910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829795.294904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829795.294905:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829795.294974:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829796.062912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829796.318919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829796.318919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829796.318949:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829797.086942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829797.342921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829797.342921:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829797.342987:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829798.110920:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829798.110928:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1787829798.110931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829798.110931:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829798.110943:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4738 00000100:00080000:1.0:1787829798.110953:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829798.110959:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829798.110992:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564442880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829798.111014:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829798.111016:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829798.111018:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829798.111021:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4738 00000100:00080000:1.0:1787829798.111023:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829798.111025:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829798.111033:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829798.111050:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564443008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829798.111059:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829798.111062:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829798.111063:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829798.111068:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564442880:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829798.111070:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4738 00000100:00080000:1.0:1787829798.111077:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829798.111082:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829798.111088:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829798.111108:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564443136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829798.111120:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829798.111126:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787829798.111126:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787829798.111131:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829798.111138:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4738 00000100:00080000:1.0:1787829798.111143:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829798.111146:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829798.111151:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829798.111162:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564443264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829798.111170:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829798.111172:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829798/real 1787829798] req@ffff976889efcb40 x1874670564442880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829798.111173:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829798.111176:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829798.111183:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4738 00000100:00080000:1.0:1787829798.111185:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829798.111186:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829798.111189:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829798.111199:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564443392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829798.111210:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564442880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829798.111220:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829798.111223:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829798.111227:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564442880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829798.111246:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564443008:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829798.111264:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829798.111273:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829798/real 1787829798] req@ffff976889efe1c0 x1874670564443008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829798.111281:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564443008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829798.111287:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829798.111289:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829798.111291:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564443008:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829798.111295:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564443136:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829798.111306:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829798.111312:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829798/real 1787829798] req@ffff976889efe580 x1874670564443136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829798.111319:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564443136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829798.111324:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829798.111326:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829798.111327:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564443136:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829798.111331:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564443264:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829798.111341:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829798.111348:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829798/real 1787829798] req@ffff976889eff480 x1874670564443264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829798.111355:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564443264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829798.111360:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829798.111362:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829798.111363:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564443264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829798.111368:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564443392:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829798.111386:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829798.111397:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829798/real 1787829798] req@ffff976889efc780 x1874670564443392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829798.111410:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564443392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829798.111419:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829798.111422:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829798.111426:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564443392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829798.366919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829798.366952:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829798.366980:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829799.134970:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829799.390946:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829799.390946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829799.390976:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829800.158915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829800.414898:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829800.414909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829800.414929:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829801.182931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829801.438930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829801.438930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829801.438993:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829802.206897:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829802.462903:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829802.462903:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829802.462928:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829803.230887:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829803.230912:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829803.230918:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829803.230922:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829803.230929:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4743 00000100:00080000:0.0:1787829803.230938:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829803.230943:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829803.230974:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564443520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829803.230995:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829803.230998:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829803.230999:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829803.231002:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4743 00000100:00080000:0.0:1787829803.231005:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829803.231007:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829803.231031:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829803.231043:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564443648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829803.231051:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829803.231053:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829803.231055:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829803.231057:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4743 00000100:00080000:0.0:1787829803.231059:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829803.231065:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829803.231065:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564443520:192.168.203.139@tcp:250:kworker.0 00000008:00100000:0.0:1787829803.231068:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829803.231079:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564443776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829803.231086:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829803.231088:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829803.231090:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829803.231093:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4743 00000100:00080000:0.0:1787829803.231095:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829803.231097:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829803.231101:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829803.231109:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564443904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829803.231117:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829803.231119:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829803.231120:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829803.231122:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4743 00000100:00080000:0.0:1787829803.231125:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829803.231128:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829803.231130:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829803.231152:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829803.231161:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564444032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829803.231188:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829803/real 1787829803] req@ffff976882aee940 x1874670564443520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829803.231218:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564443520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829803.231226:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829803.231230:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829803.231233:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564443520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829803.231239:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564443648:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829803.231261:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829803.231286:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829803/real 1787829803] req@ffff976882aeed00 x1874670564443648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829803.231300:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564443648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829803.231307:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829803.231309:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829803.231311:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564443648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829803.231331:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564443776:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829803.231346:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829803.231354:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829803/real 1787829803] req@ffff976882aec3c0 x1874670564443776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829803.231362:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564443776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829803.231372:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829803.231374:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829803.231376:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564443776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829803.231380:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564443904:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829803.231390:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829803.231396:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829803/real 1787829803] req@ffff976882aee580 x1874670564443904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829803.231403:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564443904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829803.231409:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829803.231410:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829803.231412:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564443904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829803.231416:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564444032:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829803.231425:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829803.231431:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829803/real 1787829803] req@ffff976882aed680 x1874670564444032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829803.231438:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564444032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829803.231443:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829803.231444:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829803.231447:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564444032:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829803.486919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829803.486924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829803.487047:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829804.254901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829804.510924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829804.510940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829804.510956:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829805.278944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829805.534928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829805.534951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829805.534959:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829806.302914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829806.558916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829806.558916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829806.558939:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829807.326959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829807.582937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829807.582948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829807.582982:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829808.350896:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829808.350905:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829808.350909:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829808.350911:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829808.350917:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4748 00000100:00080000:0.0:1787829808.350927:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829808.350934:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829808.350965:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564444160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829808.350986:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829808.350988:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829808.350989:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829808.350994:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4748 00000100:00080000:0.0:1787829808.350998:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829808.351002:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829808.351020:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829808.351021:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564444160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829808.351039:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564444288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829808.351052:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829808.351055:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829808.351058:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829808.351062:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4748 00000100:00080000:0.0:1787829808.351066:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829808.351069:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829808.351074:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829808.351086:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564444416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829808.351094:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829808.351102:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829808.351106:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829808.351109:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829808.351115:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4748 00000100:00080000:0.0:1787829808.351118:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829808.351123:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829808.351128:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829808.351136:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829808/real 1787829808] req@ffff976882aef0c0 x1874670564444160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829808.351142:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564444544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829808.351166:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829808.351169:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829808.351171:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829808.351174:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4748 00000100:00080000:0.0:1787829808.351178:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829808.351179:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564444160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829808.351183:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829808.351189:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829808.351200:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564444672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829808.351202:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829808.351213:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829808.351218:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564444160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829808.351242:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564444288:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829808.351268:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829808.351283:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829808/real 1787829808] req@ffff976882aed680 x1874670564444288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829808.351297:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564444288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829808.351304:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829808.351306:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829808.351309:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564444288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829808.351314:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564444416:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787829808.351330:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829808.351338:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829808/real 1787829808] req@ffff976882aee580 x1874670564444416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829808.351346:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564444416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829808.351351:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829808.351353:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829808.351356:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564444416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787829808.351359:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564444544:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829808.351370:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829808.351375:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829808/real 1787829808] req@ffff976882aec3c0 x1874670564444544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829808.351382:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564444544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829808.351388:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829808.351390:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829808.351391:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564444544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829808.351395:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564444672:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787829808.351404:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829808.351410:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829808/real 1787829808] req@ffff976882aeed00 x1874670564444672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829808.351418:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564444672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829808.351423:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829808.351424:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829808.351426:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564444672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829808.606899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829808.606902:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829808.606923:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829809.374912:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829809.630976:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829809.631017:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829809.631047:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787829810.398927:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829810.654898:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829810.654922:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829810.654928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829811.422949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829811.678937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829811.678969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829811.679023:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829812.446991:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829812.702897:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829812.702924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829812.702961:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829813.470916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829813.470977:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829813.470985:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829813.470991:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829813.471004:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4753 00000100:00080000:1.0:1787829813.471021:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829813.471030:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829813.471083:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564444800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829813.471126:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829813.471131:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829813.471133:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829813.471139:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4753 00000100:00080000:1.0:1787829813.471144:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829813.471148:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829813.471161:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829813.471175:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564444928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829813.471186:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829813.471189:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829813.471190:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829813.471194:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4753 00000100:00080000:1.0:1787829813.471196:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829813.471198:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829813.471202:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829813.471210:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564445056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829813.471217:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829813.471220:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829813.471222:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829813.471229:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4753 00000100:00080000:1.0:1787829813.471234:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829813.471237:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829813.471242:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829813.471250:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564445184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829813.471256:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829813.471258:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829813.471259:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829813.471264:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4753 00000100:00100000:3.0:1787829813.471264:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564444800:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829813.471269:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829813.471273:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829813.471276:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829813.471284:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564445312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829813.471340:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829813.471384:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829813/real 1787829813] req@ffff976889efc780 x1874670564444800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829813.471402:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564444800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829813.471411:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829813.471415:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829813.471418:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564444800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829813.471448:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564444928:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829813.471468:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829813.471481:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829813/real 1787829813] req@ffff976889eff480 x1874670564444928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829813.471496:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564444928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829813.471506:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829813.471510:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829813.471514:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564444928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829813.471519:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564445056:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829813.471553:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829813.471564:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829813/real 1787829813] req@ffff976889efe580 x1874670564445056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829813.471575:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564445056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829813.471583:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829813.471586:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829813.471589:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564445056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829813.471594:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564445184:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829813.471630:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829813.471671:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829813/real 1787829813] req@ffff976889efe1c0 x1874670564445184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829813.471699:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564445184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829813.471708:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829813.471711:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829813.471715:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564445184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829813.471722:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564445312:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829813.471744:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829813.471755:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829813/real 1787829813] req@ffff976889efcb40 x1874670564445312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829813.471763:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564445312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829813.471769:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829813.471771:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829813.471773:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564445312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829813.726943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829813.726944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829813.727124:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829814.494962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829814.750972:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829814.751013:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829814.751021:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829815.518877:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829815.774930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829815.774956:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829815.774997:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829816.542944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829816.798950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829816.798950:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829816.798979:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829817.566881:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829817.822912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829817.822913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829817.822974:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829818.590947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829818.591088:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829818.591093:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829818.591096:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829818.591103:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4758 00000100:00080000:2.0:1787829818.591118:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829818.591125:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829818.591164:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564445440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829818.591203:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829818.591208:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829818.591210:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829818.591215:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4758 00000100:00080000:2.0:1787829818.591218:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829818.591222:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829818.591231:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829818.591243:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564445568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829818.591249:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564445440:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829818.591254:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829818.591259:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829818.591260:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829818.591263:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4758 00000100:00080000:2.0:1787829818.591265:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829818.591267:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829818.591271:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829818.591287:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564445696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829818.591293:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829818.591296:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829818.591297:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829818.591300:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4758 00000100:00080000:2.0:1787829818.591303:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829818.591306:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:3.0:1787829818.591312:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:2.0:1787829818.591313:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829818.591323:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564445824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829818.591330:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829818.591332:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829818.591334:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829818.591336:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4758 00000100:00080000:2.0:1787829818.591339:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829818.591341:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829818.591344:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829818.591350:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829818/real 1787829818] req@ffff976889cfa940 x1874670564445440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829818.591354:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564445952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829818.591390:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564445440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829818.591399:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829818.591402:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829818.591405:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564445440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829818.591423:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564445568:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829818.591438:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829818.591446:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829818/real 1787829818] req@ffff976889cfb480 x1874670564445568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829818.591455:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564445568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829818.591461:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829818.591463:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829818.591465:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564445568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829818.591468:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564445696:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829818.591478:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829818.591484:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829818/real 1787829818] req@ffff976889cfad00 x1874670564445696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829818.591491:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564445696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829818.591496:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829818.591498:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829818.591500:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564445696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829818.591503:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564445824:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829818.591514:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829818.591520:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829818/real 1787829818] req@ffff976889cfa1c0 x1874670564445824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829818.591527:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564445824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829818.591532:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829818.591534:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829818.591536:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564445824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829818.591539:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564445952:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829818.591548:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829818.591553:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829818/real 1787829818] req@ffff976889cf8f00 x1874670564445952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829818.591560:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564445952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829818.591565:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829818.591567:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829818.591568:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564445952:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829818.846956:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829818.846968:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829818.846992:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829819.614916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829819.870884:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829819.870905:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829819.870939:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829820.638923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829820.894928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829820.894935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829820.894990:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829821.662908:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829821.918899:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829821.918912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829821.918960:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829822.686940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829822.942888:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829822.942902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829822.942947:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829823.710892:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829823.710893:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829823.710901:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829823.710904:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829823.710911:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4763 00000100:00080000:2.0:1787829823.710919:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829823.710925:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829823.710953:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564446080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829823.710972:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829823.710974:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829823.710976:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829823.710978:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4763 00000100:00080000:2.0:1787829823.710980:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829823.710982:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829823.710989:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829823.710998:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564446208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829823.711005:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829823.711007:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787829823.711007:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564446080:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829823.711010:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829823.711017:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4763 00000100:00080000:2.0:1787829823.711018:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829823.711020:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829823.711023:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829823.711030:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564446336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829823.711035:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829823.711038:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829823.711044:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829823.711046:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4763 00000100:00080000:2.0:1787829823.711049:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829823.711052:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829823.711055:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1787829823.711061:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829823.711064:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564446464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829823.711075:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829823.711078:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829823.711080:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829823.711084:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4763 00000100:00080000:2.0:1787829823.711087:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829823.711091:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829823.711097:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829823.711103:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829823/real 1787829823] req@ffff976889cf92c0 x1874670564446080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829823.711108:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564446592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829823.711139:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564446080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829823.711150:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829823.711153:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829823.711161:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564446080:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829823.711180:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564446208:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829823.711196:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829823.711205:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829823/real 1787829823] req@ffff976889cf8f00 x1874670564446208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829823.711213:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564446208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829823.711217:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829823.711219:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829823.711221:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564446208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829823.711225:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564446336:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829823.711234:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829823.711239:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829823/real 1787829823] req@ffff976889cfa1c0 x1874670564446336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829823.711250:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564446336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829823.711254:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829823.711256:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829823.711258:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564446336:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829823.711261:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564446464:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829823.711269:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829823.711274:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829823/real 1787829823] req@ffff976889cfad00 x1874670564446464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829823.711280:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564446464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829823.711285:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829823.711286:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829823.711288:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564446464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829823.711291:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564446592:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829823.711299:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829823.711304:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829823/real 1787829823] req@ffff976889cfb480 x1874670564446592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829823.711309:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564446592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829823.711318:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829823.711319:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829823.711321:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564446592:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829823.966894:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829823.966900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829823.966950:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829824.734917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829824.990894:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829824.990894:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829824.990921:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829825.758899:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829826.014888:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829826.014907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829826.014948:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829826.782924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829827.038919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829827.038919:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829827.039008:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829827.806934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829828.062932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829828.062932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829828.063026:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829828.830959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829828.830969:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829828.830979:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829828.830984:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829828.830998:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4768 00000100:00080000:1.0:1787829828.831013:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829828.831022:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829828.831078:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564446720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829828.831108:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829828.831111:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829828.831112:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829828.831116:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4768 00000100:00080000:1.0:1787829828.831119:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829828.831122:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829828.831133:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829828.831145:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564446848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829828.831153:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829828.831159:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564446720:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829828.831160:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829828.831164:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829828.831175:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4768 00000100:00080000:1.0:1787829828.831179:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829828.831181:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829828.831184:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829828.831193:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564446976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829828.831202:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829828.831205:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829828.831207:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829828.831213:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4768 00000100:00080000:1.0:1787829828.831217:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829828.831221:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829828.831227:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829828.831242:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564447104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829828.831244:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787829828.831255:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829828.831260:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829828.831263:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829828.831268:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4768 00000100:00080000:1.0:1787829828.831275:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829828.831281:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829828.831287:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829828.831303:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564447232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829828.831303:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829828/real 1787829828] req@ffff976889efe940 x1874670564446720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829828.831357:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564446720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829828.831369:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829828.831374:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829828.831378:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564446720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829828.831401:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564446848:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829828.831428:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829828.831440:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829828/real 1787829828] req@ffff976889efc3c0 x1874670564446848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829828.831452:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564446848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829828.831461:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829828.831464:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829828.831467:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564446848:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829828.831474:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564446976:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829828.831492:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829828.831502:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829828/real 1787829828] req@ffff976889effc00 x1874670564446976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829828.831513:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564446976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829828.831521:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829828.831537:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829828.831541:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564446976:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829828.831548:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564447104:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829828.831568:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829828.831579:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829828/real 1787829828] req@ffff976889efed00 x1874670564447104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829828.831590:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564447104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829828.831598:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829828.831601:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829828.831604:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564447104:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829828.831610:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564447232:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829828.831627:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829828.831637:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829828/real 1787829828] req@ffff976889eff0c0 x1874670564447232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829828.831648:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564447232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829828.831655:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829828.831657:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829828.831660:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564447232:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829829.086970:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829829.086971:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829829.087020:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829829.854877:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829830.110896:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829830.110921:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829830.110996:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829830.878920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829831.134935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829831.134935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829831.134992:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829831.902929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829832.158886:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829832.158944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829832.158973:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829832.926935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829833.182933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829833.182942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829833.182978:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829833.950971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829833.951018:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829833.951024:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829833.951028:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829833.951038:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4773 00000100:00080000:2.0:1787829833.951049:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829833.951058:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829833.951104:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564447360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829833.951127:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829833.951130:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829833.951131:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829833.951134:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4773 00000100:00080000:2.0:1787829833.951142:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829833.951145:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829833.951154:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829833.951166:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564447360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829833.951167:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564447488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829833.951179:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829833.951181:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829833.951182:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829833.951185:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4773 00000100:00080000:2.0:1787829833.951187:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829833.951189:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829833.951192:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829833.951198:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564447616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829833.951204:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829833.951207:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829833.951208:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829833.951210:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4773 00000100:00080000:2.0:1787829833.951222:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829833.951224:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829833.951227:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829833.951234:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564447744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829833.951238:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829833.951246:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829833.951248:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829833.951250:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829833.951253:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4773 00000100:00080000:2.0:1787829833.951256:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829833.951259:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829833.951261:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829833.951268:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564447872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829833.951289:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829833/real 1787829833] req@ffff976889cfa940 x1874670564447360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829833.951308:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564447360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829833.951319:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829833.951323:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829833.951327:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564447360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829833.951352:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564447488:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829833.951370:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829833.951381:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829833/real 1787829833] req@ffff976889cfbc00 x1874670564447488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829833.951393:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564447488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829833.951401:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829833.951404:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829833.951407:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564447488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829833.951413:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564447616:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829833.951432:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829833.951441:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829833/real 1787829833] req@ffff976889cf9e00 x1874670564447616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829833.951448:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564447616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829833.951454:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829833.951455:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829833.951457:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564447616:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829833.951461:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564447744:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829833.951473:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829833.951478:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829833/real 1787829833] req@ffff976889cfa580 x1874670564447744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829833.951503:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564447744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829833.951509:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829833.951511:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829833.951513:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564447744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829833.951517:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564447872:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829833.951530:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829833.951537:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829833/real 1787829833] req@ffff976889cfb0c0 x1874670564447872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829833.951544:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564447872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829833.951549:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829833.951551:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829833.951552:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564447872:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829834.206902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829834.206963:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829834.206998:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829834.974938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829835.230955:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829835.230966:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829835.231056:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829835.998893:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829836.254941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829836.254984:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829836.254985:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829837.022929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829837.278937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829837.278938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829837.279036:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829838.046955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829838.302951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829838.302951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829838.303001:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829839.070997:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829839.071006:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829839.071011:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829839.071011:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829839.071029:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4778 00000100:00080000:1.0:1787829839.071040:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829839.071047:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829839.071083:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564448000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829839.071117:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829839.071124:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829839.071126:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829839.071130:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4778 00000100:00080000:1.0:1787829839.071133:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829839.071137:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829839.071146:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829839.071158:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564448128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829839.071166:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564448000:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829839.071169:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829839.071174:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829839.071175:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829839.071179:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4778 00000100:00080000:1.0:1787829839.071183:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829839.071186:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829839.071191:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829839.071201:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564448256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829839.071208:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829839.071211:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829839.071213:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829839.071218:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4778 00000100:00080000:1.0:1787829839.071222:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829839.071225:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829839.071231:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829839.071242:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564448384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829839.071251:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829839.071254:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829839.071256:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829839.071260:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4778 00000100:00080000:1.0:1787829839.071263:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829839.071265:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564448128:192.168.203.139@tcp:38:kworker.0 00000100:00080000:1.0:1787829839.071268:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829839.071277:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829839.071290:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564448512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829839.071304:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564448256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829839.071321:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564448384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829839.071337:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564448512:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829839.326913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829839.326940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829839.326977:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829840.094968:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829840.350978:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829840.350978:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829840.351026:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829841.118975:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829841.374937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829841.374937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829841.374992:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829842.142959:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:0.0:1787829842.143117:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:0.0:1787829842.143171:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829842.143188:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787829842.143231:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787829842.143247:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:0.0:1787829842.143260:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:3.0:1787829842.143261:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829839/real 1787829842] req@ffff976889efed00 x1874670564448128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787829842.143276:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:3.0:1787829842.143296:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564448128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829842.143310:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829842.143316:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829842.143320:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564448128:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829842.143327:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829839/real 1787829842] req@ffff976889effc00 x1874670564448256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829842.143338:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564448256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829842.143346:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829842.143349:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829842.143352:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564448256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829842.143359:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829839/real 1787829842] req@ffff976889efc3c0 x1874670564448384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829842.143370:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564448384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829842.143384:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829842.143388:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829842.143392:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564448384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829842.143399:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829839/real 1787829842] req@ffff976889eff0c0 x1874670564448000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829842.143411:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564448000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829842.143417:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829842.143420:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829842.143423:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564448000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829842.143468:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829839/real 1787829842] req@ffff976889efe940 x1874670564448512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829842.143483:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564448512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829842.143491:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829842.143495:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829842.143498:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564448512:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829842.398971:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829842.398971:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829842.399036:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829843.166914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829843.422907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829843.422911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829843.422960:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829844.190945:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829844.190957:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829844.190962:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829844.190973:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4784 00000100:00080000:2.0:1787829844.190984:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829844.190991:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829844.191025:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564448640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829844.191052:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829844.191056:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829844.191058:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829844.191063:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4784 00000100:00080000:2.0:1787829844.191066:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829844.191071:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829844.191081:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829844.191095:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564448768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829844.191106:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564448640:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829844.191110:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829844.191117:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829844.191119:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829844.191124:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4784 00000100:00080000:2.0:1787829844.191127:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829844.191132:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829844.191137:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829844.191156:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564448896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829844.191175:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787829844.191182:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829844.191183:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829844.191187:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829844.191193:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4784 00000100:00080000:2.0:1787829844.191198:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829844.191202:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829844.191208:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829844.191221:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564449024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829844.191231:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829844.191233:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829844/real 1787829844] req@ffff976889cf8000 x1874670564448640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829844.191239:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829844.191243:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829844.191252:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4784 00000100:00080000:2.0:1787829844.191259:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829844.191266:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829844.191272:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829844.191275:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564448640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829844.191287:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564449152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829844.191290:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829844.191301:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829844.191310:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564448640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829844.191333:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:3.0:1787829844.191339:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564448768:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829844.191369:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829844.191384:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829844/real 1787829844] req@ffff976889cfb0c0 x1874670564448768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829844.191397:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564448768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829844.191406:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829844.191409:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829844.191413:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564448768:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829844.191436:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564448896:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829844.191458:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829844.191470:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829844/real 1787829844] req@ffff976889cfa580 x1874670564448896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829844.191483:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564448896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829844.191498:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829844.191501:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829844.191505:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564448896:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829844.191511:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564449024:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829844.191530:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829844.191541:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829844/real 1787829844] req@ffff976889cf9e00 x1874670564449024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829844.191554:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564449024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829844.191562:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829844.191566:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829844.191569:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564449024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829844.191575:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564449152:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829844.191594:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829844.191604:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829844/real 1787829844] req@ffff976889cfbc00 x1874670564449152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829844.191616:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564449152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829844.191624:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829844.191627:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829844.191631:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564449152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829844.446929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829844.446929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829844.446998:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829845.214936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829845.470927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829845.470932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829845.470984:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829846.238939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829846.494949:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829846.494955:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829846.495045:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829847.262893:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829847.518926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829847.518926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829847.519017:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829848.286954:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829848.542927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829848.542927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829848.542997:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829849.310902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829849.310952:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829849.310956:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829849.310959:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829849.310970:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4789 00000100:00080000:3.0:1787829849.310980:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829849.310988:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829849.311023:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564449280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829849.311050:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829849.311054:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829849.311055:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829849.311067:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4789 00000100:00080000:3.0:1787829849.311071:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829849.311076:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829849.311086:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829849.311087:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564449280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829849.311103:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564449408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829849.311115:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829849.311119:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829849.311121:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829849.311126:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4789 00000100:00080000:3.0:1787829849.311132:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829849.311137:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829849.311141:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829849.311155:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564449536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829849.311160:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787829849.311169:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829849.311171:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829849.311173:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829849.311177:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4789 00000100:00080000:3.0:1787829849.311180:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829849.311183:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829849.311186:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829849.311195:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564449664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829849.311202:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829849.311206:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829849.311208:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829849/real 1787829849] req@ffff97688cb5f840 x1874670564449280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829849.311208:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829849.311215:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4789 00000100:00080000:3.0:1787829849.311220:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829849.311235:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564449280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829849.311246:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829849.311251:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829849.311255:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564449280:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787829849.311282:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829849.311284:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564449408:192.168.203.139@tcp:38:kworker.0 00000008:00100000:3.0:1787829849.311287:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829849.311301:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564449792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829849.311316:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829849.311336:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829849/real 1787829849] req@ffff97688cb5d680 x1874670564449408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829849.311349:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564449408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829849.311358:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829849.311361:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829849.311365:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564449408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829849.311371:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564449536:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829849.311405:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829849.311417:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829849/real 1787829849] req@ffff97688cb5de00 x1874670564449536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829849.311437:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564449536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829849.311446:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829849.311449:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829849.311453:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564449536:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829849.311462:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564449664:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829849.311487:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829849.311503:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829849/real 1787829849] req@ffff97688cb5cb40 x1874670564449664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829849.311518:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564449664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829849.311528:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829849.311533:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829849.311537:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564449664:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829849.311568:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564449792:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829849.311606:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829849.311618:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829849/real 1787829849] req@ffff97688cb5c000 x1874670564449792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829849.311626:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564449792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829849.311637:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829849.311639:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829849.311641:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564449792:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829849.566903:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829849.566908:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829849.567078:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829850.334920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829850.590932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829850.590947:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829850.590953:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787829851.358947:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829851.614898:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829851.614898:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829851.614969:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829852.382881:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829852.638918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829852.638919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829852.638949:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829853.406921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829853.662917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829853.662918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829853.662958:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829854.430951:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829854.430960:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829854.430964:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829854.430967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829854.430974:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4794 00000100:00080000:1.0:1787829854.430984:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829854.430991:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829854.431025:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564449920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829854.431055:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829854.431059:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829854.431062:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829854.431068:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4794 00000100:00080000:1.0:1787829854.431074:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829854.431079:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829854.431089:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829854.431105:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564449920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829854.431112:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564450048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829854.431124:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829854.431127:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829854.431128:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829854.431131:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4794 00000100:00080000:1.0:1787829854.431133:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829854.431137:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829854.431141:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829854.431151:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564450176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829854.431157:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829854.431160:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829854.431161:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829854.431164:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4794 00000100:00080000:1.0:1787829854.431166:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829854.431168:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829854.431172:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829854.431185:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564450304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829854.431197:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787829854.431208:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829854.431211:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829854.431213:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829854.431217:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4794 00000100:00080000:1.0:1787829854.431228:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829854.431232:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829854.431235:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829854.431246:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564450432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829854.431253:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829854/real 1787829854] req@ffff976889efcb40 x1874670564449920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829854.431278:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564449920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829854.431288:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829854.431292:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829854.431295:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564449920:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829854.431321:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564450048:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829854.431340:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829854.431375:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829854/real 1787829854] req@ffff976889efe1c0 x1874670564450048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829854.431389:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564450048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829854.431398:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829854.431402:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829854.431407:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564450048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829854.431414:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564450176:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829854.431436:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829854.431447:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829854/real 1787829854] req@ffff976889efe580 x1874670564450176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829854.431455:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564450176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829854.431461:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829854.431464:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829854.431466:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564450176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829854.431471:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564450304:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829854.431485:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829854.431492:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829854/real 1787829854] req@ffff976889eff480 x1874670564450304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829854.431499:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564450304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829854.431505:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829854.431508:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829854.431510:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564450304:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829854.431514:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564450432:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829854.431524:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829854.431531:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829854/real 1787829854] req@ffff976889efc780 x1874670564450432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829854.431538:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564450432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829854.431543:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829854.431545:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829854.431547:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564450432:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829854.686934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829854.686934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829854.686966:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829855.454962:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829855.711010:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829855.711012:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829855.711035:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829856.478934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829856.734924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829856.734926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829856.734990:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829857.502929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829857.758951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829857.758984:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829857.759018:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829858.526924:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829858.782915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829858.782915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829858.783001:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829859.550964:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829859.550990:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829859.551000:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829859.551006:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829859.551020:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4799 00000100:00080000:1.0:1787829859.551038:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829859.551047:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829859.551096:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564450560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829859.551130:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829859.551134:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829859.551136:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829859.551141:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4799 00000100:00080000:1.0:1787829859.551146:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829859.551151:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829859.551163:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564450560:192.168.203.139@tcp:250:kworker.0 00000008:00100000:1.0:1787829859.551165:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829859.551185:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564450688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829859.551198:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829859.551202:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829859.551204:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829859.551209:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4799 00000100:00080000:1.0:1787829859.551213:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829859.551216:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829859.551222:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829859.551235:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564450816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829859.551239:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787829859.551250:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829859.551254:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829859.551257:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829859.551262:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4799 00000100:00080000:1.0:1787829859.551266:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829859.551271:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829859.551277:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829859.551288:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829859/real 1787829859] req@ffff976889efc780 x1874670564450560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829859.551293:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564450944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829859.551323:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829859.551328:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564450560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829859.551329:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829859.551334:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829859.551344:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4799 00000100:00080000:3.0:1787829859.551347:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829859.551349:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829859.551353:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:1.0:1787829859.551358:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829859.551360:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564450560:192.168.203.139@tcp:250:kworker.0 00000008:00100000:1.0:1787829859.551368:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829859.551382:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564451072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829859.551391:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564450688:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829859.551415:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829859.551426:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829859/real 1787829859] req@ffff976889eff480 x1874670564450688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829859.551434:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564450688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829859.551441:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829859.551443:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829859.551446:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564450688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829859.551449:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564450816:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829859.551466:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829859.551477:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829859/real 1787829859] req@ffff976889efe580 x1874670564450816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829859.551491:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564450816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829859.551500:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829859.551503:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829859.551507:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564450816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829859.551513:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564450944:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829859.551532:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829859.551543:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829859/real 1787829859] req@ffff976889efe1c0 x1874670564450944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829859.551555:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564450944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829859.551563:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829859.551565:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829859.551569:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564450944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829859.551588:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564451072:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829859.551608:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829859.551620:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829859/real 1787829859] req@ffff976889efcb40 x1874670564451072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829859.551633:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564451072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829859.551642:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829859.551645:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829859.551648:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564451072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829859.806937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829859.806987:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829859.806991:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829860.574914:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829860.830911:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829860.830912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829860.830963:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829861.598938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829861.854957:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829861.854959:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829861.855040:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829862.622897:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829862.878915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829862.878919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829862.878983:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829863.646926:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829863.902937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829863.902937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829863.902998:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829864.670910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829864.670921:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829864.670927:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829864.670931:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829864.670939:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4804 00000100:00080000:3.0:1787829864.670948:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829864.670955:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829864.670993:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564451200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829864.671022:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829864.671025:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829864.671027:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829864.671031:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4804 00000100:00080000:3.0:1787829864.671036:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829864.671040:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829864.671048:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829864.671054:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564451200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829864.671063:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564451328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829864.671076:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829864.671080:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829864.671082:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829864.671086:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4804 00000100:00080000:3.0:1787829864.671091:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829864.671096:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829864.671102:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829864.671112:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829864.671117:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564451456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829864.671126:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829864.671128:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829864.671130:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829864.671133:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4804 00000100:00080000:3.0:1787829864.671137:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829864.671140:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829864.671145:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829864.671151:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829864/real 1787829864] req@ffff97688cb5c000 x1874670564451200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829864.671156:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564451584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829864.671177:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829864.671179:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829864.671183:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564451200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829864.671183:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829864.671191:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4804 00000100:00080000:3.0:1787829864.671195:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829864.671200:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:1.0:1787829864.671204:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829864.671205:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829864.671213:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829864.671219:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564451200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829864.671219:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564451712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829864.671248:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564451328:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829864.671274:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829864.671296:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829864/real 1787829864] req@ffff97688cb5cb40 x1874670564451328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829864.671305:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564451328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829864.671312:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829864.671314:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829864.671316:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564451328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829864.671320:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564451456:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829864.671333:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829864.671339:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829864/real 1787829864] req@ffff97688cb5de00 x1874670564451456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829864.671347:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564451456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829864.671352:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829864.671354:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829864.671356:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564451456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829864.671360:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564451584:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829864.671370:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829864.671376:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829864/real 1787829864] req@ffff97688cb5d680 x1874670564451584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829864.671383:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564451584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829864.671393:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829864.671395:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829864.671397:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564451584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829864.671400:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564451712:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829864.671409:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829864.671415:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829864/real 1787829864] req@ffff97688cb5f840 x1874670564451712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829864.671421:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564451712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829864.671426:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829864.671428:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829864.671429:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564451712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829864.926915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829864.926915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829864.926971:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829865.694952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829865.950950:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829865.950977:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829865.951051:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829866.718902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829866.974937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829866.974937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829866.975006:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829867.742923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829867.998929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829867.998934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829867.999010:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829868.766985:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829869.022940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829869.022941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829869.023008:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829869.790954:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829869.790961:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829869.790971:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829869.790977:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829869.790989:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4809 00000100:00080000:0.0:1787829869.791004:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829869.791014:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829869.791076:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564451840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829869.791108:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829869.791111:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829869.791112:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829869.791116:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4809 00000100:00080000:0.0:1787829869.791119:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829869.791123:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829869.791134:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829869.791148:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564451968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829869.791156:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829869.791159:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829869.791160:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829869.791164:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4809 00000100:00080000:0.0:1787829869.791168:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829869.791170:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564451840:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829869.791171:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829869.791176:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829869.791186:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564452096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829869.791192:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829869.791195:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829869.791196:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829869.791199:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4809 00000100:00080000:0.0:1787829869.791201:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829869.791204:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829869.791207:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829869.791214:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564452224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829869.791221:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829869.791224:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829869.791225:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829869.791228:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4809 00000100:00080000:0.0:1787829869.791262:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829869.791265:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829869.791269:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829869.791276:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829869.791280:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564452352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829869.791330:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829869/real 1787829869] req@ffff976882aee940 x1874670564451840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829869.791351:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564451840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829869.791365:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829869.791370:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829869.791374:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564451840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829869.791401:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564451968:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829869.791425:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829869.791438:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829869/real 1787829869] req@ffff976882aeed00 x1874670564451968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829869.791452:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564451968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829869.791462:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829869.791465:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829869.791469:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564451968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829869.791476:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564452096:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829869.791497:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829869.791509:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829869/real 1787829869] req@ffff976882aec3c0 x1874670564452096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829869.791520:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564452096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829869.791528:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829869.791540:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829869.791544:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564452096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829869.791551:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564452224:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829869.791570:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829869.791581:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829869/real 1787829869] req@ffff976882aee580 x1874670564452224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829869.791592:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564452224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829869.791600:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829869.791604:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829869.791607:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564452224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829869.791613:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564452352:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829869.791629:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829869.791638:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829869/real 1787829869] req@ffff976882aed680 x1874670564452352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829869.791648:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564452352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829869.791656:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829869.791659:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829869.791662:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564452352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829870.046950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829870.046952:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829870.046983:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829870.815039:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829871.070932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829871.070945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829871.070996:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829871.838981:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829872.094963:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829872.094963:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829872.095006:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829872.862949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829873.118927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829873.118928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829873.119006:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829873.886949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829874.142936:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829874.142941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829874.142974:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829874.910946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829874.910969:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829874.910978:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829874.910983:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829874.910995:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4814 00000100:00080000:2.0:1787829874.911007:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829874.911014:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829874.911058:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564452480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829874.911089:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829874.911093:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829874.911095:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829874.911100:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4814 00000100:00080000:2.0:1787829874.911104:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829874.911109:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829874.911121:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829874.911135:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564452608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829874.911145:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829874.911149:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787829874.911151:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564452480:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829874.911151:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829874.911156:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4814 00000100:00080000:2.0:1787829874.911159:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829874.911163:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829874.911168:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829874.911177:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564452736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829874.911185:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829874.911188:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829874.911190:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829874.911194:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4814 00000100:00080000:2.0:1787829874.911197:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829874.911201:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829874.911220:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829874.911231:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564452864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829874.911240:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829874.911243:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829874.911244:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829874.911246:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829874.911249:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4814 00000100:00080000:2.0:1787829874.911252:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829874.911256:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829874.911260:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829874.911269:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564452992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829874.911294:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829874/real 1787829874] req@ffff976889cfa940 x1874670564452480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829874.911312:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564452480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829874.911325:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829874.911330:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829874.911333:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564452480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829874.911358:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564452608:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829874.911378:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829874.911388:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829874/real 1787829874] req@ffff976889cfb480 x1874670564452608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829874.911396:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564452608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829874.911402:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829874.911405:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829874.911407:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564452608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829874.911410:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564452736:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829874.911421:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829874.911428:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829874/real 1787829874] req@ffff976889cfad00 x1874670564452736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829874.911436:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564452736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829874.911442:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829874.911444:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829874.911445:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564452736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829874.911449:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564452864:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829874.911461:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829874.911471:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829874/real 1787829874] req@ffff976889cfa1c0 x1874670564452864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829874.911484:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564452864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829874.911493:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829874.911497:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829874.911500:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564452864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829874.911506:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564452992:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829874.911525:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829874.911534:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829874/real 1787829874] req@ffff976889cf8f00 x1874670564452992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829874.911542:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564452992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829874.911552:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829874.911555:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829874.911557:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564452992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829875.166968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829875.166969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829875.167045:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829875.934906:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829876.190926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829876.190942:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829876.190942:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829876.958928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829877.214944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829877.214947:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829877.215025:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829877.982979:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829878.238948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829878.238949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829878.239022:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829879.006953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829879.262958:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829879.262964:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829879.263024:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829880.030973:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829880.030973:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829880.030987:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829880.031002:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829880.031016:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4819 00000100:00080000:0.0:1787829880.031030:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829880.031039:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829880.031091:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564453120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829880.031131:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829880.031135:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829880.031137:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829880.031142:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4819 00000100:00080000:0.0:1787829880.031146:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829880.031151:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829880.031163:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829880.031192:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564453248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829880.031206:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829880.031208:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564453120:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829880.031212:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829880.031216:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829880.031222:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4819 00000100:00080000:0.0:1787829880.031227:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829880.031231:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829880.031238:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829880.031251:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564453376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829880.031262:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829880.031266:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829880.031269:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829880.031274:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4819 00000100:00080000:0.0:1787829880.031285:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829880.031290:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829880.031296:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1787829880.031297:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829880.031312:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564453504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829880.031324:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829880.031328:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829880.031331:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829880.031335:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4819 00000100:00080000:0.0:1787829880.031339:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829880.031344:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829880.031350:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829880.031370:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564453632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829880.031371:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829880/real 1787829880] req@ffff976882aef0c0 x1874670564453120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829880.031412:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564453120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829880.031425:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829880.031430:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829880.031435:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564453120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829880.031463:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564453248:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829880.031491:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829880.031508:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829880/real 1787829880] req@ffff976882aed680 x1874670564453248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829880.031531:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564453248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829880.031541:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829880.031544:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829880.031548:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564453248:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829880.031556:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564453376:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829880.031579:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829880.031593:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829880/real 1787829880] req@ffff976882aee580 x1874670564453376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829880.031606:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564453376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829880.031616:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829880.031620:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829880.031623:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564453376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829880.031631:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564453504:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829880.031654:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829880.031667:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829880/real 1787829880] req@ffff976882aec3c0 x1874670564453504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829880.031680:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564453504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829880.031697:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829880.031700:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829880.031704:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564453504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829880.031711:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564453632:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829880.031730:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829880.031739:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829880/real 1787829880] req@ffff976882aeed00 x1874670564453632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829880.031746:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564453632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829880.031752:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829880.031755:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829880.031757:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564453632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829880.286915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829880.286915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829880.286988:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829881.054961:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829881.310964:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829881.310973:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829881.310993:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829882.078958:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829882.334956:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829882.334956:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829882.335165:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829883.103029:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829883.358990:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829883.358991:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829883.359009:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829884.126932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829884.382961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829884.382961:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829884.383057:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829885.150948:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829885.151042:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829885.151051:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829885.151055:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829885.151065:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4825 00000100:00080000:0.0:1787829885.151077:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829885.151086:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829885.151161:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564453760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829885.151241:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564453760:192.168.203.139@tcp:250:kworker.0 00000800:00000100:1.0:1787829885.151330:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829885.151371:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829885.151375:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829885.151378:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829885.151383:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4825 00000100:00080000:0.0:1787829885.151388:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829885.151394:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829885.151414:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829885.151433:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564453888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829885.151498:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829885/real 1787829885] req@ffff976882aee940 x1874670564453760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829885.151517:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564453760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829885.151530:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829885.151534:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829885.151539:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564453760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829885.151566:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564453888:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829885.151591:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829885.151605:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829885/real 1787829885] req@ffff976882aeed00 x1874670564453888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829885.151618:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564453888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829885.151627:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829885.151630:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829885.151634:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564453888:192.168.203.139@tcp:38:kworker.0 00000100:00080000:0.0:1787829885.151701:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829885.151706:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829885.151712:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829885.151717:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4825 00000100:00080000:0.0:1787829885.151722:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829885.151727:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829885.151735:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829885.151750:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564454016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829885.151769:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829885.151773:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829885.151776:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829885.151782:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4825 00000100:00080000:0.0:1787829885.151787:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829885.151790:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564454016:192.168.203.139@tcp:38:kworker.0 00000100:00080000:0.0:1787829885.151794:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829885.151804:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829885.151818:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564454144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829885.151853:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829885.151867:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829885.151871:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829885.151876:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829885.151877:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829885/real 1787829885] req@ffff976882aec3c0 x1874670564454016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829885.151886:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4825 00000100:00080000:0.0:1787829885.151894:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829885.151901:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829885.151906:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564454016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829885.151910:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829885.151928:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564454272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829885.151934:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829885.151946:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829885.151952:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564454016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829885.151970:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564454144:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829885.151997:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829885.152014:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829885/real 1787829885] req@ffff976882aee580 x1874670564454144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829885.152030:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564454144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829885.152042:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829885.152047:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829885.152052:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564454144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829885.152060:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564454272:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829885.152087:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829885.152102:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829885/real 1787829885] req@ffff976882aed680 x1874670564454272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829885.152118:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564454272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829885.152126:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829885.152131:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829885.152150:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564454272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829885.406970:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829885.406978:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829885.407003:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829886.174915:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829886.430983:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829886.431010:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829886.431060:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829887.198933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829887.454940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829887.454972:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829887.454996:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829888.222893:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829888.478889:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829888.478893:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829888.478939:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829889.246997:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829889.502894:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829889.502900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829889.502940:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829890.270937:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829890.270939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829890.270947:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829890.270951:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829890.270959:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4830 00000100:00080000:0.0:1787829890.270970:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829890.270976:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829890.271005:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564454400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829890.271028:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829890.271031:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829890.271032:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829890.271035:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4830 00000100:00080000:0.0:1787829890.271038:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829890.271040:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829890.271051:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829890.271062:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564454528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829890.271069:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829890.271072:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829890.271073:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829890.271076:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4830 00000100:00080000:0.0:1787829890.271078:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829890.271081:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829890.271084:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829890.271093:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564454656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829890.271095:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564454400:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829890.271121:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829890.271125:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829890.271126:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829890.271130:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4830 00000100:00080000:0.0:1787829890.271132:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829890.271135:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829890.271139:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829890.271149:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564454784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829890.271157:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829890.271159:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829890.271160:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829890.271163:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4830 00000100:00080000:0.0:1787829890.271171:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829890.271174:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829890.271177:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829890.271183:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829890.271186:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564454912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829890.271232:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829890/real 1787829890] req@ffff976882aef0c0 x1874670564454400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829890.271253:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564454400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829890.271265:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829890.271269:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829890.271274:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564454400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829890.271283:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564454528:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829890.271310:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829890.271326:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829890/real 1787829890] req@ffff976882aed680 x1874670564454528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829890.271349:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564454528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829890.271360:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829890.271364:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829890.271368:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564454528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829890.271395:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564454656:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829890.271419:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829890.271433:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829890/real 1787829890] req@ffff976882aee580 x1874670564454656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829890.271458:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564454656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829890.271468:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829890.271471:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829890.271475:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564454656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829890.271482:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564454784:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829890.271503:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829890.271515:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829890/real 1787829890] req@ffff976882aec3c0 x1874670564454784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829890.271527:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564454784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829890.271535:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829890.271548:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829890.271552:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564454784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829890.271559:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564454912:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829890.271581:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829890.271592:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829890/real 1787829890] req@ffff976882aeed00 x1874670564454912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829890.271605:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564454912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829890.271623:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829890.271626:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829890.271629:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564454912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829890.526894:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829890.526899:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829890.526955:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829891.295120:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829891.550902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829891.550920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829891.550986:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829892.318933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829892.574901:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829892.574904:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829892.574958:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829893.342928:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829893.598910:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829893.598951:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829893.598968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829894.366971:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829894.622932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829894.622938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829894.623002:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829895.390909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829895.390915:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829895.390921:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829895.390924:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829895.390932:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4835 00000100:00080000:3.0:1787829895.390941:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829895.390946:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829895.390977:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564455040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829895.390998:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829895.391001:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829895.391003:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829895.391007:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4835 00000100:00080000:3.0:1787829895.391011:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829895.391016:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829895.391026:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829895.391038:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564455168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829895.391049:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829895.391052:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829895.391055:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829895.391056:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564455040:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787829895.391059:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4835 00000100:00080000:3.0:1787829895.391075:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829895.391079:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829895.391085:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829895.391095:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564455296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829895.391105:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829895.391108:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829895.391110:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829895.391114:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4835 00000100:00080000:3.0:1787829895.391117:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829895.391120:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829895.391124:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1787829895.391128:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829895.391133:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564455424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829895.391142:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829895.391145:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829895.391146:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829895.391150:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4835 00000100:00080000:3.0:1787829895.391153:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829895.391156:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829895.391160:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829895.391171:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564455552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829895.391174:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829895/real 1787829895] req@ffff97688cb5e1c0 x1874670564455040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829895.391191:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564455040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829895.391202:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829895.391206:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829895.391210:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564455040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787829895.391237:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564455168:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829895.391262:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829895.391275:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829895/real 1787829895] req@ffff97688cb5f0c0 x1874670564455168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829895.391284:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564455168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829895.391291:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829895.391293:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829895.391295:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564455168:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829895.391301:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564455296:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829895.391321:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829895.391332:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829895/real 1787829895] req@ffff97688cb5c780 x1874670564455296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829895.391344:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564455296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829895.391353:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829895.391356:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829895.391359:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564455296:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829895.391366:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564455424:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829895.391386:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829895.391394:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829895/real 1787829895] req@ffff97688cb5d2c0 x1874670564455424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829895.391402:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564455424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829895.391407:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829895.391410:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829895.391412:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564455424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829895.391416:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564455552:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829895.391427:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829895.391433:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829895/real 1787829895] req@ffff97688cb5c3c0 x1874670564455552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829895.391440:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564455552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829895.391445:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829895.391447:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829895.391449:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564455552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829895.646933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829895.646933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829895.646995:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829896.414934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829896.670945:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829896.670960:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829896.670977:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829897.438923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829897.694941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829897.694941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829897.695002:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829898.462935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829898.718935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829898.718949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829898.718978:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829899.486910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829899.742900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829899.742913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829899.742962:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829900.510921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829900.510933:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829900.510939:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829900.510944:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829900.510952:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4840 00000100:00080000:0.0:1787829900.510960:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829900.510977:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829900.511008:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564455680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829900.511039:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829900.511042:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829900.511044:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829900.511048:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4840 00000100:00080000:0.0:1787829900.511051:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829900.511055:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829900.511063:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829900.511076:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564455680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829900.511077:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564455808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829900.511089:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829900.511092:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829900.511094:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829900.511097:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4840 00000100:00080000:0.0:1787829900.511100:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829900.511104:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829900.511109:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829900.511118:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564455936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829900.511126:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829900.511131:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829900.511135:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829900.511137:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829900.511142:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4840 00000100:00080000:0.0:1787829900.511145:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829900.511149:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829900.511154:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829900.511159:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829900/real 1787829900] req@ffff976882aee940 x1874670564455680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829900.511165:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564456064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829900.511174:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564455680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829900.511176:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829900.511185:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829900.511187:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829900.511191:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829900.511192:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4840 00000100:00080000:1.0:1787829900.511197:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:0.0:1787829900.511198:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829900.511203:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564455680:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829900.511204:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829900.511211:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829900.511221:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564456192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829900.511224:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564455808:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829900.511245:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829900.511256:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829900/real 1787829900] req@ffff976882aeed00 x1874670564455808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829900.511264:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564455808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829900.511269:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829900.511271:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829900.511273:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564455808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829900.511276:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564455936:192.168.203.139@tcp:38:kworker.0 00000800:00000100:1.0:1787829900.511285:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829900.511290:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829900/real 1787829900] req@ffff976882aec3c0 x1874670564455936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829900.511296:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564455936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829900.511300:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829900.511302:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829900.511304:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564455936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829900.511307:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564456064:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829900.511316:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829900.511321:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829900/real 1787829900] req@ffff976882aee580 x1874670564456064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829900.511327:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564456064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829900.511332:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829900.511333:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829900.511335:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564456064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829900.511344:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564456192:192.168.203.139@tcp:8:kworker.0 00000800:00000100:1.0:1787829900.511352:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829900.511357:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829900/real 1787829900] req@ffff976882aed680 x1874670564456192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829900.511364:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564456192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829900.511368:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829900.511370:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829900.511371:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564456192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829900.766919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829900.766919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829900.766947:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829901.534890:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829901.790935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829901.790935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829901.791090:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829902.558922:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829902.814908:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829902.814908:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829902.814932:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829903.582890:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829903.838943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829903.838947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829903.839031:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829904.606921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829904.862897:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829904.862903:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829904.862921:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829905.630885:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829905.630922:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829905.630926:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829905.630929:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829905.630936:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4845 00000100:00080000:0.0:1787829905.630944:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829905.630949:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829905.630986:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564456320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829905.631023:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829905.631026:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829905.631028:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829905.631032:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4845 00000100:00080000:0.0:1787829905.631036:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829905.631040:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829905.631048:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829905.631059:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564456448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829905.631067:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829905.631070:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829905.631072:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829905.631072:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564456320:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829905.631077:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4845 00000100:00080000:0.0:1787829905.631082:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829905.631085:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829905.631088:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829905.631096:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564456576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829905.631108:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829905.631110:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829905.631111:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829905.631113:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4845 00000100:00080000:0.0:1787829905.631123:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829905.631125:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829905.631128:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829905.631134:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564456704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829905.631140:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829905.631142:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829905.631143:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829905.631145:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4845 00000100:00080000:0.0:1787829905.631146:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829905.631148:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829905.631151:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829905.631159:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564456832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829905.631171:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564456448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829905.631202:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564456576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829905.631220:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564456704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829905.631243:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564456832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829905.886909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829905.886909:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829905.886940:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829906.654999:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829906.910920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829906.910920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829906.910949:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829907.679001:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829907.934908:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829907.934970:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829907.935037:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787829908.703036:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:1.0:1787829908.703136:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:1.0:1787829908.703173:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787829908.703193:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787829908.703228:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787829908.703238:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787829908.703247:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787829908.703255:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:3.0:1787829908.703273:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829905/real 1787829908] req@ffff976882aeed00 x1874670564456832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829908.703295:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564456832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829908.703316:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829908.703321:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829908.703325:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564456832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829908.703331:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829905/real 1787829908] req@ffff976882aed680 x1874670564456448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829908.703343:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564456448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829908.703350:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829908.703353:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829908.703356:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564456448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829908.703362:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829905/real 1787829908] req@ffff976882aee580 x1874670564456576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829908.703373:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564456576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829908.703381:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829908.703383:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829908.703386:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564456576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829908.703391:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829905/real 1787829908] req@ffff976882aec3c0 x1874670564456704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829908.703401:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564456704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829908.703408:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829908.703414:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829908.703416:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564456704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829908.703422:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829905/real 1787829908] req@ffff976882aef0c0 x1874670564456320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829908.703433:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564456320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829908.703438:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829908.703441:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829908.703445:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564456320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829908.958920:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829908.958920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829908.958952:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829909.727026:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829909.982906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829909.982909:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829909.982954:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829910.750917:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829910.750927:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829910.750931:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829910.750940:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4850 00000100:00080000:0.0:1787829910.750950:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829910.750967:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829910.750977:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829910.750999:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564456960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829910.751026:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829910.751029:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829910.751035:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829910.751038:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4850 00000100:00080000:0.0:1787829910.751042:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829910.751045:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829910.751061:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829910.751074:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564457088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829910.751080:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564456960:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829910.751086:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829910.751090:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829910.751092:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829910.751096:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4850 00000100:00080000:0.0:1787829910.751100:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829910.751104:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829910.751108:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829910.751119:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564457216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829910.751127:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829910.751130:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829910.751132:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829910.751136:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4850 00000100:00080000:0.0:1787829910.751139:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829910.751144:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829910.751145:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829910.751151:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829910.751163:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564457344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829910.751171:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829910.751174:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829910.751176:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829910.751180:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4850 00000100:00080000:0.0:1787829910.751184:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829910.751188:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829910/real 1787829910] req@ffff976882aee940 x1874670564456960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829910.751190:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829910.751197:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829910.751212:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564457472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829910.751216:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564456960/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829910.751245:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829910.751250:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829910.751254:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564456960:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829910.751277:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564457088:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829910.751299:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829910.751310:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829910/real 1787829910] req@ffff976882aef0c0 x1874670564457088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829910.751322:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564457088/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829910.751330:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829910.751334:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829910.751337:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564457088:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829910.751342:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564457216:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829910.751358:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829910.751368:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829910/real 1787829910] req@ffff976882aec3c0 x1874670564457216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829910.751384:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564457216/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829910.751392:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829910.751395:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829910.751398:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564457216:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829910.751403:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564457344:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829910.751419:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829910.751429:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829910/real 1787829910] req@ffff976882aee580 x1874670564457344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829910.751440:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564457344/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829910.751447:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829910.751451:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829910.751454:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564457344:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829910.751460:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564457472:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829910.751476:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829910.751485:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829910/real 1787829910] req@ffff976882aed680 x1874670564457472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829910.751495:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564457472/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829910.751508:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829910.751511:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829910.751514:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564457472:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829911.006926:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829911.006926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829911.006965:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829911.774996:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829912.030920:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829912.030976:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829912.030981:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829912.798936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829913.054907:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829913.054907:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829913.054937:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829913.822957:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829914.078924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829914.078924:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829914.078954:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829914.846892:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829915.102914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829915.102914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829915.102950:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829915.870875:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829915.870876:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829915.870883:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829915.870886:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829915.870893:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4855 00000100:00080000:0.0:1787829915.870906:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829915.870911:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829915.870953:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564457600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829915.870978:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829915.870982:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829915.870984:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829915.870988:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4855 00000100:00080000:0.0:1787829915.870992:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829915.870996:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829915.871005:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829915.871018:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564457600:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829915.871021:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564457728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829915.871032:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829915.871036:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829915.871038:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829915.871042:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4855 00000100:00080000:0.0:1787829915.871045:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829915.871056:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829915.871061:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829915.871077:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564457856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829915.871077:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829915.871089:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829915.871093:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829915.871095:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829915.871099:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4855 00000100:00080000:0.0:1787829915.871102:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829915.871107:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829915.871112:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829915.871115:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829915/real 1787829915] req@ffff976882aeed00 x1874670564457600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829915.871126:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564457984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829915.871131:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564457600/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829915.871150:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829915.871151:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829915.871157:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829915.871158:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:0.0:1787829915.871161:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829915.871163:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564457600:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829915.871169:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4855 00000100:00080000:0.0:1787829915.871173:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829915.871176:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829915.871180:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829915.871191:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564458112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829915.871193:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564457728:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829915.871218:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829915.871233:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829915/real 1787829915] req@ffff976882aed680 x1874670564457728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829915.871240:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564457728/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829915.871246:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829915.871248:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829915.871250:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564457728:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829915.871254:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564457856:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829915.871264:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829915.871270:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829915/real 1787829915] req@ffff976882aee580 x1874670564457856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829915.871281:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564457856/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829915.871286:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829915.871288:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829915.871290:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564457856:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829915.871293:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564457984:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829915.871302:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829915.871307:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829915/real 1787829915] req@ffff976882aec3c0 x1874670564457984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829915.871313:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564457984/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829915.871322:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829915.871324:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829915.871326:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564457984:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829915.871337:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564458112:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829915.871351:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829915.871357:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829915/real 1787829915] req@ffff976882aef0c0 x1874670564458112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829915.871363:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564458112/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829915.871368:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829915.871370:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829915.871371:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564458112:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829916.126947:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829916.126975:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829916.126987:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829916.894938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829917.150901:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829917.150927:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829917.150942:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829917.918931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829918.174929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829918.174932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829918.174970:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829918.942931:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829919.198888:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829919.198891:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829919.198895:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829919.966946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829920.222930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829920.222930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829920.222988:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829920.990935:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829920.990977:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829920.990981:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829920.990984:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829920.990991:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4860 00000100:00080000:3.0:1787829920.990999:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829920.991004:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829920.991032:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564458240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829920.991047:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829920.991052:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829920.991054:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829920.991057:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4860 00000100:00080000:3.0:1787829920.991060:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829920.991062:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829920.991070:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829920.991078:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564458240:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829920.991082:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564458368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829920.991092:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829920.991094:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829920.991095:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829920.991098:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4860 00000100:00080000:3.0:1787829920.991100:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829920.991103:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829920.991106:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829920.991115:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564458496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829920.991122:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829920.991124:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829920.991126:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829920.991129:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4860 00000100:00080000:3.0:1787829920.991132:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829920.991135:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829920.991139:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1787829920.991145:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829920.991148:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564458624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829920.991154:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829920.991156:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829920.991158:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829920.991185:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829920/real 1787829920] req@ffff97688cb5c3c0 x1874670564458240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829920.991202:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564458240/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829920.991209:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4860 00000100:00080000:0.0:1787829920.991211:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829920.991215:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829920.991217:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:3.0:1787829920.991221:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829920.991223:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564458240:192.168.203.139@tcp:250:kworker.0 00000008:00100000:3.0:1787829920.991227:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829920.991237:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564458752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829920.991251:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564458368:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829920.991270:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829920.991279:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829920/real 1787829920] req@ffff97688cb5d2c0 x1874670564458368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829920.991287:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564458368/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829920.991292:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829920.991295:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829920.991297:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564458368:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829920.991301:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564458496:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829920.991312:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829920.991319:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829920/real 1787829920] req@ffff97688cb5c780 x1874670564458496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829920.991328:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564458496/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829920.991337:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829920.991342:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829920.991346:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564458496:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829920.991353:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564458624:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829920.991377:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829920.991389:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829920/real 1787829920] req@ffff97688cb5f0c0 x1874670564458624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829920.991397:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564458624/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829920.991402:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829920.991404:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829920.991405:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564458624:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829920.991409:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564458752:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829920.991421:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829920.991427:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829920/real 1787829920] req@ffff97688cb5e1c0 x1874670564458752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829920.991433:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564458752/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829920.991438:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829920.991439:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829920.991441:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564458752:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829921.246917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829921.246918:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829921.246947:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829922.014932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829922.270936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829922.270976:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829922.271026:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829923.038904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829923.294976:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829923.294981:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829923.295004:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829924.062921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829924.318925:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829924.318992:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829924.319013:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787829925.086942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829925.342941:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829925.342941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829925.342972:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829926.110934:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829926.110941:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829926.110944:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829926.110950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829926.110952:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4866 00000100:00080000:2.0:1787829926.110962:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829926.110968:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829926.110996:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564458880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829926.111018:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829926.111020:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829926.111021:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829926.111023:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4866 00000100:00080000:2.0:1787829926.111025:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829926.111027:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829926.111034:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829926.111043:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564459008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829926.111050:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829926.111054:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787829926.111056:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564458880:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829926.111057:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829926.111066:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4866 00000100:00080000:2.0:1787829926.111071:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829926.111075:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829926.111079:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829926.111093:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564459136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829926.111102:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829926.111105:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829926.111108:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829926.111111:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829926.111116:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4866 00000100:00080000:2.0:1787829926.111121:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829926.111126:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829926.111130:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829926.111139:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564459264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829926.111144:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829926.111146:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829926.111148:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829926.111150:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4866 00000100:00080000:2.0:1787829926.111153:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829926.111155:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829926/real 1787829926] req@ffff976889cf92c0 x1874670564458880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829926.111158:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829926.111162:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829926.111172:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564459392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829926.111179:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564458880/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829926.111187:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829926.111190:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829926.111193:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564458880:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829926.111209:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564459008:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829926.111225:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829926.111233:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829926/real 1787829926] req@ffff976889cf8f00 x1874670564459008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829926.111240:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564459008/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829926.111255:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829926.111258:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829926.111260:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564459008:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829926.111263:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564459136:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829926.111274:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829926.111280:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829926/real 1787829926] req@ffff976889cfa1c0 x1874670564459136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829926.111286:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564459136/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829926.111295:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829926.111297:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829926.111299:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564459136:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829926.111302:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564459264:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829926.111312:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829926.111317:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829926/real 1787829926] req@ffff976889cfad00 x1874670564459264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829926.111332:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564459264/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829926.111337:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829926.111338:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829926.111340:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564459264:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829926.111343:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564459392:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829926.111352:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829926.111357:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829926/real 1787829926] req@ffff976889cfb480 x1874670564459392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829926.111363:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564459392/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829926.111367:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829926.111369:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829926.111370:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564459392:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829926.366969:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829926.366982:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829926.367058:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829927.134896:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829927.390933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829927.390946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829927.390963:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829928.158913:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829928.414940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829928.414940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829928.414993:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829929.182892:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829929.438915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829929.438915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829929.438939:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829930.206923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829930.462930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829930.462954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829930.462986:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829931.230923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829931.230925:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829931.230934:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829931.230938:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829931.230947:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4871 00000100:00080000:0.0:1787829931.230964:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829931.230972:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829931.231006:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564459520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829931.231038:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829931.231041:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829931.231043:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829931.231047:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4871 00000100:00080000:0.0:1787829931.231051:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829931.231055:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829931.231065:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829931.231071:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564459520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829931.231081:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564459648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829931.231093:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829931.231096:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829931.231098:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829931.231102:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4871 00000100:00080000:0.0:1787829931.231105:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829931.231109:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829931.231113:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829931.231123:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564459776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829931.231131:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829931.231143:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829931.231147:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829931.231149:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829931.231153:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4871 00000100:00080000:0.0:1787829931.231157:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829931.231160:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829931.231165:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829931.231172:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829931/real 1787829931] req@ffff976882aee940 x1874670564459520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829931.231178:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564459904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829931.231196:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829931.231198:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829931.231202:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829931.231202:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564459520/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829931.231208:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4871 00000100:00080000:0.0:1787829931.231211:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829931.231216:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829931.231223:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787829931.231229:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829931.231235:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564460032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829931.231236:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829931.231245:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564459520:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829931.231272:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564459648:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829931.231293:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829931.231304:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829931/real 1787829931] req@ffff976882aef0c0 x1874670564459648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829931.231316:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564459648/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829931.231324:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829931.231327:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829931.231330:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564459648:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829931.231336:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564459776:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829931.231352:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829931.231361:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829931/real 1787829931] req@ffff976882aec3c0 x1874670564459776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829931.231387:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564459776/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829931.231394:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829931.231397:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829931.231400:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564459776:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829931.231405:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564459904:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829931.231422:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829931.231432:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829931/real 1787829931] req@ffff976882aee580 x1874670564459904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829931.231443:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564459904/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829931.231450:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829931.231452:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829931.231455:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564459904:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829931.231460:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564460032:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829931.231475:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829931.231484:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829931/real 1787829931] req@ffff976882aed680 x1874670564460032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829931.231494:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564460032/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829931.231508:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829931.231511:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829931.231514:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564460032:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829931.486896:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829931.486906:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829931.486956:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829932.255069:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829932.510938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829932.510938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829932.510976:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829933.278952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829933.534918:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829933.534924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829933.534942:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829934.302930:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829934.558930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829934.558965:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829934.559000:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829935.327983:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829935.582930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829935.582930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829935.582999:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829936.350918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829936.350952:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829936.350960:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829936.350965:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829936.350974:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4876 00000100:00080000:0.0:1787829936.350988:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829936.350995:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829936.351036:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564460160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829936.351058:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829936.351061:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829936.351062:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829936.351065:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4876 00000100:00080000:0.0:1787829936.351067:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829936.351070:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829936.351079:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829936.351089:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564460288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829936.351097:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829936.351101:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829936.351104:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829936.351105:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564460160:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829936.351109:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4876 00000100:00080000:0.0:1787829936.351114:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829936.351123:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829936.351127:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829936.351136:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564460416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829936.351144:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829936.351146:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829936.351147:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829936.351150:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4876 00000100:00080000:0.0:1787829936.351152:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829936.351154:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829936.351158:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829936.351165:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564460544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829936.351172:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829936.351175:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829936.351177:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829936.351182:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4876 00000100:00080000:0.0:1787829936.351185:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829936.351189:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829936.351196:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787829936.351196:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829936.351215:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564460672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829936.351251:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829936/real 1787829936] req@ffff976882aeed00 x1874670564460160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829936.351273:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564460160/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829936.351285:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829936.351289:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829936.351293:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564460160:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829936.351319:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564460288:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829936.351343:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829936.351355:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829936/real 1787829936] req@ffff976882aed680 x1874670564460288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829936.351373:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564460288/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829936.351381:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829936.351385:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829936.351388:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564460288:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829936.351394:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564460416:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829936.351413:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829936.351423:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829936/real 1787829936] req@ffff976882aee580 x1874670564460416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829936.351435:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564460416/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829936.351443:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829936.351446:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829936.351450:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564460416:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829936.351456:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564460544:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829936.351476:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829936.351486:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829936/real 1787829936] req@ffff976882aec3c0 x1874670564460544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829936.351497:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564460544/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829936.351510:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829936.351513:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829936.351516:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564460544:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829936.351522:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564460672:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829936.351539:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829936.351548:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829936/real 1787829936] req@ffff976882aef0c0 x1874670564460672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829936.351559:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564460672/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829936.351566:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829936.351569:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829936.351572:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564460672:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829936.606917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829936.606940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829936.606968:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829937.374916:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829937.630966:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829937.630983:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829937.630986:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829938.398940:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829938.654896:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829938.654938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829938.654951:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829939.422918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829939.678949:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829939.678949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829939.678996:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829940.446969:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829940.702960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829940.703008:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829940.703023:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00080000:2.0:1787829941.470951:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829941.470961:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829941.470966:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829941.470975:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829941.470975:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4881 00000100:00080000:2.0:1787829941.470993:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829941.471002:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829941.471039:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564460800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829941.471071:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829941.471074:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829941.471076:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829941.471080:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4881 00000100:00080000:2.0:1787829941.471084:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829941.471088:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829941.471097:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829941.471109:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564460928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829941.471119:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564460800:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829941.471122:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829941.471129:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829941.471130:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829941.471135:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4881 00000100:00080000:2.0:1787829941.471139:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829941.471142:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829941.471147:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829941.471160:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564461056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829941.471169:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829941.471173:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829941.471174:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829941.471178:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4881 00000800:00000100:3.0:1787829941.471182:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829941.471183:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829941.471190:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829941.471195:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787829941.471207:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564461184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829941.471216:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829941.471219:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829941.471220:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829941.471225:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829941/real 1787829941] req@ffff976889cfa940 x1874670564460800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829941.471227:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4881 00000100:00080000:2.0:1787829941.471233:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829941.471240:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829941.471246:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829941.471250:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564460800/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829941.471260:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564461312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829941.471265:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829941.471271:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829941.471276:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564460800:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829941.471297:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564460928:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829941.471313:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829941.471321:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829941/real 1787829941] req@ffff976889cfbc00 x1874670564460928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829941.471329:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564460928/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829941.471334:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829941.471335:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829941.471337:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564460928:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829941.471341:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564461056:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829941.471350:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829941.471355:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829941/real 1787829941] req@ffff976889cf9e00 x1874670564461056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829941.471361:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564461056/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829941.471366:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829941.471367:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829941.471369:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564461056:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829941.471373:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564461184:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829941.471383:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829941.471387:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829941/real 1787829941] req@ffff976889cfa580 x1874670564461184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829941.471394:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564461184/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829941.471398:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829941.471400:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829941.471401:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564461184:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829941.471404:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564461312:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829941.471411:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829941.471416:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829941/real 1787829941] req@ffff976889cfb0c0 x1874670564461312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829941.471430:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564461312/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829941.471435:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829941.471436:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829941.471438:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564461312:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829941.726913:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829941.726927:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829941.726937:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829942.494966:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829942.750914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829942.750914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829942.750952:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829943.518939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829943.774912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829943.774915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829943.774935:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829944.542956:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829944.798934:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829944.798968:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829944.798989:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787829945.566923:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829945.822916:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829945.822916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829945.822946:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829946.590878:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829946.590885:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829946.590889:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829946.590899:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4886 00000100:00080000:0.0:1787829946.590908:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829946.590909:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829946.590916:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829946.590947:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564461440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829946.590966:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829946.590968:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829946.590969:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829946.590972:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4886 00000100:00080000:0.0:1787829946.590974:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829946.590978:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829946.590986:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829946.590999:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564461568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829946.591004:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564461440:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829946.591017:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829946.591020:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829946.591021:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829946.591025:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4886 00000100:00080000:0.0:1787829946.591027:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829946.591030:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829946.591034:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829946.591041:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564461696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829946.591048:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829946.591050:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829946.591051:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829946.591053:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4886 00000100:00080000:0.0:1787829946.591056:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829946.591058:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829946.591062:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829946.591074:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564461824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829946.591074:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829946.591086:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829946.591089:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829946.591092:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829946.591096:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4886 00000100:00080000:0.0:1787829946.591100:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829946.591104:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829946.591107:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829946/real 1787829946] req@ffff976882aee940 x1874670564461440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829946.591118:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829946.591122:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564461440/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829946.591131:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564461952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829946.591131:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829946.591138:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829946.591144:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564461440:192.168.203.139@tcp:250:kworker.0 00000100:00100000:3.0:1787829946.591164:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564461568:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829946.591200:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829946.591208:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829946/real 1787829946] req@ffff976882aef0c0 x1874670564461568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829946.591217:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564461568/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829946.591225:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829946.591228:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829946.591231:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564461568:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829946.591236:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564461696:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829946.591255:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829946.591265:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829946/real 1787829946] req@ffff976882aec3c0 x1874670564461696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829946.591274:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564461696/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829946.591279:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829946.591281:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829946.591282:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564461696:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829946.591310:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564461824:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829946.591325:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829946.591332:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829946/real 1787829946] req@ffff976882aee580 x1874670564461824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829946.591343:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564461824/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829946.591350:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829946.591352:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829946.591355:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564461824:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829946.591361:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564461952:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829946.591378:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829946.591387:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829946/real 1787829946] req@ffff976882aed680 x1874670564461952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829946.591399:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564461952/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829946.591403:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829946.591405:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829946.591407:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564461952:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829946.846931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829946.846932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829946.847002:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829947.614920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:3.0:1787829947.870957:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829947.870959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829947.871041:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787829948.638917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829948.894914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829948.894914:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829948.894991:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829949.662895:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829949.918909:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829949.918912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829949.918943:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829950.686921:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829950.942912:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829950.942937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829950.943053:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829951.710893:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787829951.710901:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829951.710909:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829951.710914:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829951.710924:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4891 00000100:00080000:0.0:1787829951.710934:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829951.710941:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829951.710976:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aeed00 x1874670564462080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829951.711006:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829951.711010:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829951.711012:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829951.711016:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4891 00000100:00080000:0.0:1787829951.711019:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829951.711023:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829951.711032:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829951.711047:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564462208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829951.711052:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564462080:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829951.711067:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829951.711071:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829951.711073:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829951.711078:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4891 00000100:00080000:0.0:1787829951.711083:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829951.711087:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829951.711092:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829951.711104:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564462336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829951.711113:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:0.0:1787829951.711126:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829951.711131:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829951.711133:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829951.711140:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4891 00000100:00080000:0.0:1787829951.711145:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829951.711150:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829951/real 1787829951] req@ffff976882aeed00 x1874670564462080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829951.711151:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787829951.711160:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829951.711171:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aeed00 x1874670564462080/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829951.711177:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564462464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829951.711183:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829951.711193:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:0.0:1787829951.711199:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829951.711200:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aeed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564462080:192.168.203.139@tcp:250:kworker.0 00000100:00080000:0.0:1787829951.711208:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829951.711212:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829951.711217:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4891 00000100:00080000:0.0:1787829951.711221:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829951.711225:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829951.711229:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564462208:192.168.203.139@tcp:38:kworker.0 00000008:00100000:0.0:1787829951.711232:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829951.711249:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564462592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829951.711258:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829951.711270:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829951/real 1787829951] req@ffff976882aed680 x1874670564462208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829951.711285:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564462208/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829951.711293:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829951.711296:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829951.711298:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564462208:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829951.711302:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564462336:192.168.203.139@tcp:38:kworker.0 00000800:00000100:3.0:1787829951.711313:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829951.711320:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829951/real 1787829951] req@ffff976882aee580 x1874670564462336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829951.711327:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564462336/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829951.711332:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829951.711334:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829951.711336:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564462336:192.168.203.139@tcp:38:kworker.0 00000100:00100000:3.0:1787829951.711339:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564462464:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829951.711349:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829951.711355:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829951/real 1787829951] req@ffff976882aec3c0 x1874670564462464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829951.711367:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564462464/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829951.711372:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829951.711374:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829951.711375:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564462464:192.168.203.139@tcp:8:kworker.0 00000100:00100000:3.0:1787829951.711386:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564462592:192.168.203.139@tcp:8:kworker.0 00000800:00000100:3.0:1787829951.711396:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:3.0:1787829951.711402:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829951/real 1787829951] req@ffff976882aef0c0 x1874670564462592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829951.711409:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564462592/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829951.711414:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829951.711416:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:3.0:1787829951.711418:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564462592:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829951.966929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829951.966930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829951.966951:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829952.734938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829952.990988:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829952.990997:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829952.991027:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829953.758904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829954.014935:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829954.014935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:3.0:1787829954.014957:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829954.782967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829955.038899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829955.038919:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829955.038957:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829955.806993:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829956.062906:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829956.062945:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829956.063046:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787829956.830929:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829956.830950:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829956.830957:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829956.830962:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829956.830971:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4896 00000100:00080000:1.0:1787829956.830982:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829956.830989:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829956.831025:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564462720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829956.831058:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829956.831062:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829956.831064:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829956.831069:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4896 00000100:00080000:1.0:1787829956.831074:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829956.831079:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829956.831091:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829956.831107:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564462848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829956.831112:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564462720:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829956.831126:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829956.831131:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829956.831133:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829956.831137:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4896 00000100:00080000:1.0:1787829956.831141:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829956.831145:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829956.831149:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829956.831163:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564462976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829956.831179:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829956.831182:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829956.831184:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829956.831190:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787829956.831195:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4896 00000100:00080000:1.0:1787829956.831202:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829956.831207:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829956.831212:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829956.831225:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564463104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829956.831236:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829956.831238:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829956/real 1787829956] req@ffff976889efe940 x1874670564462720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829956.831242:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829956.831246:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829956.831255:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4896 00000100:00080000:1.0:1787829956.831260:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829956.831264:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829956.831266:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564462720/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829956.831270:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829956.831282:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829956.831286:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:1.0:1787829956.831290:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564463232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829956.831294:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564462720:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829956.831326:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564462848:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829956.831354:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829956.831379:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829956/real 1787829956] req@ffff976889eff0c0 x1874670564462848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829956.831393:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564462848/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829956.831402:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829956.831405:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829956.831410:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564462848:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829956.831416:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564462976:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829956.831436:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829956.831450:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829956/real 1787829956] req@ffff976889efc3c0 x1874670564462976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829956.831465:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564462976/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829956.831474:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829956.831478:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829956.831482:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564462976:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829956.831490:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564463104:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829956.831508:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829956.831519:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829956/real 1787829956] req@ffff976889effc00 x1874670564463104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829956.831529:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564463104/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829956.831537:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829956.831540:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829956.831543:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564463104:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829956.831548:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564463232:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829956.831564:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829956.831573:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829956/real 1787829956] req@ffff976889efed00 x1874670564463232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829956.831584:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564463232/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829956.831592:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829956.831595:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829956.831598:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564463232:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829957.086915:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829957.086928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829957.086998:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829957.854944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829958.110883:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829958.110912:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829958.110936:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829958.878901:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829959.134900:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829959.134914:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829959.134961:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829959.902955:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829960.158887:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829960.158925:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:3.0:1787829960.158933:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829960.926939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829961.182949:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829961.182949:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829961.184096:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829961.950914:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829961.950919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829961.950922:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829961.950928:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829961.950935:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4901 00000100:00080000:1.0:1787829961.950943:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829961.950948:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829961.950978:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564463360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829961.950999:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829961.951002:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829961.951003:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829961.951006:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4901 00000100:00080000:1.0:1787829961.951008:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829961.951010:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829961.951016:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829961.951023:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564463488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829961.951029:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829961.951032:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829961.951033:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829961.951035:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4901 00000100:00080000:1.0:1787829961.951037:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829961.951039:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829961.951042:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829961.951050:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564463616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829961.951055:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564463360:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829961.951073:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829961.951077:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829961.951079:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829961.951084:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4901 00000100:00080000:1.0:1787829961.951088:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829961.951092:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829961.951097:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787829961.951106:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564463744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829961.951115:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829961.951118:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829961.951120:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829961.951125:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787829961.951128:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4901 00000100:00080000:1.0:1787829961.951135:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829961.951140:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829961.951144:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829961.951156:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564463872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829961.951163:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829961/real 1787829961] req@ffff976889efed00 x1874670564463360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829961.951176:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564463360/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829961.951183:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829961.951187:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829961.951190:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564463360:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829961.951194:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564463488:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829961.951209:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829961.951217:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829961/real 1787829961] req@ffff976889effc00 x1874670564463488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829961.951224:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564463488/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829961.951229:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829961.951231:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829961.951233:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564463488:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829961.951250:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564463616:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829961.951259:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829961.951264:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829961/real 1787829961] req@ffff976889efc3c0 x1874670564463616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829961.951270:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564463616/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829961.951275:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829961.951277:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829961.951279:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564463616:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829961.951282:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564463744:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829961.951292:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829961.951300:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829961/real 1787829961] req@ffff976889eff0c0 x1874670564463744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829961.951310:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564463744/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829961.951318:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829961.951321:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829961.951324:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564463744:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829961.951330:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564463872:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829961.951347:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829961.951357:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829961/real 1787829961] req@ffff976889efe940 x1874670564463872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829961.951368:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564463872/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829961.951376:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829961.951386:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829961.951389:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564463872:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829962.206932:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829962.206932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829962.206979:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829962.974944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829963.230909:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829963.230915:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829963.230957:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829963.998934:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829964.254902:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829964.254902:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829964.254948:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829965.022945:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829965.278909:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829965.279594:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829965.279633:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829966.046949:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829966.302943:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829966.303003:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829966.303172:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00080000:1.0:1787829967.070899:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829967.070899:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787829967.070908:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829967.070913:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829967.070922:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4906 00000100:00080000:1.0:1787829967.070930:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829967.070938:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787829967.070980:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564464000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829967.071000:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829967.071002:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829967.071004:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829967.071007:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4906 00000100:00080000:1.0:1787829967.071009:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829967.071012:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829967.071019:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829967.071028:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564464128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829967.071036:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829967.071037:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564464000:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787829967.071038:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829967.071041:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829967.071047:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4906 00000100:00080000:1.0:1787829967.071050:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829967.071051:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829967.071054:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829967.071066:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564464256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829967.071080:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829967.071083:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829967.071085:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829967.071089:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4906 00000100:00080000:1.0:1787829967.071093:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829967.071096:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829967.071100:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1787829967.071109:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:1.0:1787829967.071114:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564464384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829967.071123:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829967.071126:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829967.071128:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829967.071131:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4906 00000100:00080000:1.0:1787829967.071143:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829967.071148:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787829967.071154:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829967.071155:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829967/real 1787829967] req@ffff976889efcb40 x1874670564464000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829967.071165:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564464512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829967.071193:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564464000/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829967.071205:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829967.071209:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829967.071212:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564464000:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829967.071239:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564464128:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829967.071258:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829967.071267:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829967/real 1787829967] req@ffff976889efe1c0 x1874670564464128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829967.071275:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564464128/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829967.071287:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829967.071290:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829967.071293:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564464128:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829967.071299:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564464256:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829967.071317:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829967.071328:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829967/real 1787829967] req@ffff976889efe580 x1874670564464256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829967.071338:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564464256/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829967.071351:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829967.071354:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829967.071357:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564464256:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829967.071363:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564464384:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829967.071382:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829967.071393:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829967/real 1787829967] req@ffff976889eff480 x1874670564464384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829967.071404:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564464384/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829967.071411:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829967.071414:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829967.071418:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564464384:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829967.071423:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564464512:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829967.071440:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829967.071448:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829967/real 1787829967] req@ffff976889efc780 x1874670564464512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829967.071454:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564464512/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829967.071459:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829967.071461:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829967.071463:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564464512:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829967.326932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829967.326948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829967.327028:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829968.094936:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829968.351016:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829968.351174:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829968.351793:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787829969.118920:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829969.374950:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829969.375053:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829969.375065:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829970.143036:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829970.398931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829970.398960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829970.399068:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829971.166991:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829971.422991:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829971.422993:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829971.423076:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829972.191017:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829972.191024:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829972.191028:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829972.191036:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4912 00000100:00080000:3.0:1787829972.191044:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829972.191050:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829972.191083:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564464640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829972.191097:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829972.191199:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564464640:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787829972.191311:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829972.191315:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829972.191317:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829972.191322:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4912 00000100:00080000:3.0:1787829972.191326:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829972.191331:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829972.191339:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829972.191352:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564464768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829972.191655:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564464768:192.168.203.139@tcp:38:kworker.0 00000100:00080000:3.0:1787829972.191732:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829972.191736:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829972.191738:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829972.191743:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4912 00000100:00080000:3.0:1787829972.191747:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829972.191752:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829972.191757:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829972.191771:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564464896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829972.191782:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829972.191785:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829972.191787:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829972.191791:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4912 00000100:00080000:3.0:1787829972.191795:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829972.191799:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829972.191804:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829972.191814:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564465024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829972.191849:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829972.191853:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829972.191855:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829972.191858:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4912 00000100:00080000:3.0:1787829972.191861:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829972.191865:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829972.191869:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829972.191880:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564465152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829972.192252:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564464896:192.168.203.139@tcp:38:kworker.0 00000100:00100000:1.0:1787829972.192282:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564465024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829972.192305:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564465152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829972.446932:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829972.446934:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829972.447275:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829973.215763:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829973.470938:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829973.470951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829973.471256:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829974.239037:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829974.494951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829974.494951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829974.495032:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829975.263420:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000400:00000100:1.0:1787829975.263565:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.139:988 00000400:00000100:1.0:1787829975.263608:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.139@tcp at host 192.168.203.139 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787829975.263623:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787829975.263663:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787829975.263672:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787829975.263680:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000800:00000100:1.0:1787829975.263688:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.139@tcp 00000100:00100000:0.0:1787829975.263740:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829972/real 1787829975] req@ffff97688cb5de00 x1874670564464896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829975.263769:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564464896/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829975.263784:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829975.263788:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829975.263791:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564464896:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829975.263797:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829972/real 1787829975] req@ffff97688cb5cb40 x1874670564465024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829975.264837:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564465024/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829975.264854:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829975.264861:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829975.264866:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564465024:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829975.264875:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829972/real 1787829975] req@ffff97688cb5c000 x1874670564465152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829975.264894:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564465152/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829975.264903:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829975.264907:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829975.264910:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564465152:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829975.264916:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829972/real 1787829975] req@ffff97688cb5d680 x1874670564464768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829975.264929:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564464768/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829975.264937:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829975.264941:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829975.264944:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564464768:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829975.264951:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829972/real 1787829975] req@ffff97688cb5f840 x1874670564464640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829975.264964:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564464640/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829975.264972:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829975.264975:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829975.264979:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564464640:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829975.518958:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829975.519334:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829975.519423:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829976.286978:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829976.542928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829976.543056:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829976.543553:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829977.310954:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829977.310967:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829977.310968:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829977.310976:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829977.310987:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4917 00000100:00080000:3.0:1787829977.310999:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829977.311007:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829977.311063:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f840 x1874670564465280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829977.311100:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829977.311104:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829977.311105:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829977.311109:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4917 00000100:00080000:3.0:1787829977.311120:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829977.311125:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829977.311135:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829977.311157:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d680 x1874670564465408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829977.311166:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829977.311176:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829977.311178:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829977.311181:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4917 00000100:00080000:3.0:1787829977.311184:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829977.311187:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829977.311191:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829977.311231:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c000 x1874670564465536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829977.311241:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829977.311243:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829977.311245:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829977.311248:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4917 00000100:00080000:3.0:1787829977.311251:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829977.311254:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829977.311259:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829977.311269:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5cb40 x1874670564465664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829977.311276:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829977.311279:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829977.311280:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829977.311283:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4917 00000100:00080000:3.0:1787829977.311286:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829977.311289:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829977.311293:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829977.311302:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5de00 x1874670564465792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829977.311613:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564465280:192.168.203.139@tcp:250:kworker.0 00000800:00000100:0.0:1787829977.311677:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829977.311713:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829977/real 1787829977] req@ffff97688cb5f840 x1874670564465280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829977.311727:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f840 x1874670564465280/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829977.311737:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829977.311740:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829977.311743:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564465280:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829977.311754:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564465408:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829977.311774:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829977.311786:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829977/real 1787829977] req@ffff97688cb5d680 x1874670564465408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829977.311803:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d680 x1874670564465408/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829977.311811:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829977.311813:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829977.311817:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564465408:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829977.311822:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564465536:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829977.311861:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829977.311872:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829977/real 1787829977] req@ffff97688cb5c000 x1874670564465536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829977.311883:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c000 x1874670564465536/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829977.311890:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829977.311892:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829977.311908:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564465536:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829977.311915:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564465664:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829977.311934:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829977.311945:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829977/real 1787829977] req@ffff97688cb5cb40 x1874670564465664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829977.311955:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5cb40 x1874670564465664/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829977.311968:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829977.311971:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829977.311974:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564465664:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829977.311980:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564465792:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829977.312001:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829977.312011:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829977/real 1787829977] req@ffff97688cb5de00 x1874670564465792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829977.312020:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5de00 x1874670564465792/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829977.312028:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829977.312031:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829977.312033:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564465792:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829977.566968:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829977.566968:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829977.568815:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829978.335222:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829978.590940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829978.590940:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829978.590972:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829979.358950:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829979.614948:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829979.614981:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829979.615143:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829980.382933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829980.638931:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829980.638931:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829980.639020:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829981.407499:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829981.662954:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829981.662960:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829981.664753:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829982.430976:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829982.430987:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829982.430991:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829982.431001:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4922 00000100:00080000:2.0:1787829982.431012:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829982.431019:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829982.431054:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8000 x1874670564465920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829982.431074:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829982.431076:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829982.431077:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829982.431081:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4922 00000100:00080000:2.0:1787829982.431084:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829982.431087:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829982.431097:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829982.431102:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829982.431113:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564466048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829982.431126:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829982.431130:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829982.431132:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829982.431136:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4922 00000100:00080000:2.0:1787829982.431139:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829982.431143:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829982.431147:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829982.431158:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564466176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829982.431166:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829982.431169:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829982.431170:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829982.431172:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564465920:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829982.431176:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4922 00000100:00080000:2.0:1787829982.431183:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829982.431187:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829982.431192:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1787829982.431229:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829982.431258:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829982/real 1787829982] req@ffff976889cf8000 x1874670564465920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829982.431269:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8000 x1874670564465920/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829982.431276:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829982.431278:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787829982.431280:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564466304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829982.431281:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564465920:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829982.431295:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829982.431299:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787829982.431302:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564466048:192.168.203.139@tcp:38:kworker.0 00000100:00080000:2.0:1787829982.431303:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829982.431311:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4922 00000100:00080000:2.0:1787829982.431316:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829982.431320:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829982.431324:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829982.431330:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829982/real 1787829982] req@ffff976889cfb0c0 x1874670564466048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829982.431331:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829982.431343:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564466048/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829982.431349:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787829982.431351:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564466432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829982.431354:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829982.431359:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564466048:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829982.431368:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564466176:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829982.431383:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829982.431391:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829982/real 1787829982] req@ffff976889cfa580 x1874670564466176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829982.431402:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564466176/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829982.431407:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829982.431408:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829982.431410:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564466176:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829982.431413:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564466304:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829982.431422:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829982.431427:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829982/real 1787829982] req@ffff976889cf9e00 x1874670564466304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829982.431433:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564466304/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829982.431437:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829982.431438:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829982.431440:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564466304:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829982.431448:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564466432:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829982.431478:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829982.431490:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829982/real 1787829982] req@ffff976889cfbc00 x1874670564466432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829982.431500:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564466432/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829982.431588:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829982.431591:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829982.431593:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564466432:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829982.686952:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829982.687319:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829982.688307:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829983.454994:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829983.710975:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829983.710979:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829983.711028:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829984.478952:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829984.734928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829984.734946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829984.734997:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829985.503099:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829985.758948:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829985.759051:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829985.759267:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829986.527101:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829986.782995:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829986.783024:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:1.0:1787829986.783221:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829987.550992:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829987.551001:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829987.551006:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829987.551016:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4927 00000100:00100000:0.0:1787829987.551023:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829987.551026:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829987.551036:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829987.551074:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564466560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829987.551101:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829987.551105:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829987.551106:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829987.551111:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4927 00000100:00080000:3.0:1787829987.551114:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829987.551117:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829987.551127:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829987.551139:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564466688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829987.551149:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829987.551152:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829987.551154:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829987.551159:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4927 00000100:00080000:3.0:1787829987.551163:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829987.551169:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564466560:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787829987.551170:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829987.551178:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829987.551191:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564466816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829987.551202:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829987.551205:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829987.551207:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829987.551211:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4927 00000100:00080000:3.0:1787829987.551215:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829987.551218:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829987.551223:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829987.551231:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564466944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829987.551240:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829987.551243:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829987.551245:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829987.551248:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4927 00000100:00080000:3.0:1787829987.551251:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829987.551254:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829987.551266:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829987.551278:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564467072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829987.551317:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829987.551373:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829987/real 1787829987] req@ffff97688cb5e1c0 x1874670564466560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829987.551389:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564466560/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829987.551398:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829987.551401:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829987.551404:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564466560:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829987.551415:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564466688:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829987.551469:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829987.551483:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829987/real 1787829987] req@ffff97688cb5f0c0 x1874670564466688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829987.551492:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564466688/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829987.551498:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829987.551500:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829987.551502:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564466688:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829987.551533:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564466816:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829987.551547:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829987.551554:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829987/real 1787829987] req@ffff97688cb5c780 x1874670564466816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829987.551561:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564466816/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829987.551566:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829987.551569:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829987.551570:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564466816:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829987.551574:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564466944:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829987.551584:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829987.551590:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829987/real 1787829987] req@ffff97688cb5d2c0 x1874670564466944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829987.551597:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564466944/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829987.551602:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829987.551605:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829987.551607:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564466944:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829987.551610:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564467072:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829987.551620:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829987.551624:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829987/real 1787829987] req@ffff97688cb5c3c0 x1874670564467072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829987.551630:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564467072/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829987.551634:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829987.551636:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829987.551638:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564467072:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829987.807153:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829987.807591:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829987.807604:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829988.575181:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829988.830961:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829988.830995:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787829988.832583:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829989.598953:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787829989.854930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829989.854930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829989.854995:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829990.623000:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829990.879023:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829990.879487:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829990.879564:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787829991.647511:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829991.902917:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829991.903039:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787829991.903090:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829992.670947:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829992.670960:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829992.670966:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829992.670982:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4932 00000100:00080000:3.0:1787829992.670995:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829992.671003:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:3.0:1787829992.671046:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c3c0 x1874670564467200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829992.671054:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:3.0:1787829992.671075:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829992.671079:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829992.671081:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829992.671086:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4932 00000100:00080000:3.0:1787829992.671091:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829992.671092:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564467200:192.168.203.139@tcp:250:kworker.0 00000100:00080000:3.0:1787829992.671097:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829992.671111:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829992.671124:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5d2c0 x1874670564467328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829992.671134:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829992.671137:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829992.671139:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829992.671143:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4932 00000100:00080000:3.0:1787829992.671146:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829992.671149:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829992.671155:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829992.671166:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5c780 x1874670564467456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829992.671176:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829992.671180:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829992.671182:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829992.671187:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:3.0:1787829992.671189:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4932 00000100:00080000:3.0:1787829992.671194:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829992.671198:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:3.0:1787829992.671203:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1787829992.671228:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5f0c0 x1874670564467584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829992.671238:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829992.671239:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829992/real 1787829992] req@ffff97688cb5c3c0 x1874670564467200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829992.671244:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829992.671248:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829992.671258:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4932 00000100:00080000:3.0:1787829992.671263:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829992.671267:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829992.671272:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c3c0 x1874670564467200/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829992.671274:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829992.671290:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829992.671295:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff97688cb5e1c0 x1874670564467712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829992.671298:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829992.671306:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564467200:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787829992.671338:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564467328:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829992.671366:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829992.671394:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829992/real 1787829992] req@ffff97688cb5d2c0 x1874670564467328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829992.671413:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5d2c0 x1874670564467328/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829992.671423:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829992.671426:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829992.671430:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564467328:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829992.671436:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564467456:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829992.671460:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829992.671474:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829992/real 1787829992] req@ffff97688cb5c780 x1874670564467456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829992.671498:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5c780 x1874670564467456/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829992.671512:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829992.671516:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829992.671520:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564467456:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829992.671526:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564467584:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829992.671542:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829992.671549:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829992/real 1787829992] req@ffff97688cb5f0c0 x1874670564467584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829992.671560:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5f0c0 x1874670564467584/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829992.671568:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829992.671572:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829992.671575:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564467584:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829992.671581:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564467712:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829992.671599:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829992.671610:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829992/real 1787829992] req@ffff97688cb5e1c0 x1874670564467712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829992.671622:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff97688cb5e1c0 x1874670564467712/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829992.671634:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829992.671637:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829992.671642:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff97688cb5e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564467712:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787829992.926902:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829992.926952:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829992.926981:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829993.695051:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829993.950938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829993.950951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787829993.951088:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829994.719008:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829994.974928:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829994.974928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829994.975009:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829995.743008:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829995.998888:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787829995.998890:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787829995.998954:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829996.766917:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787829997.022917:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787829997.022924:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829997.022982:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829997.790918:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787829997.790951:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829997.790957:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829997.790961:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829997.790970:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4937 00000100:00080000:2.0:1787829997.790979:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829997.790984:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787829997.791013:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564467840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829997.791032:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829997.791035:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829997.791036:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829997.791038:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4937 00000100:00080000:2.0:1787829997.791041:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829997.791043:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829997.791051:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829997.791062:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564467840:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787829997.791062:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564467968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829997.791076:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829997.791079:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829997.791081:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829997.791085:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4937 00000100:00080000:2.0:1787829997.791089:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829997.791093:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829997.791098:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829997.791109:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564468096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829997.791118:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787829997.791120:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829997.791124:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829997.791126:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829997.791132:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4937 00000100:00080000:2.0:1787829997.791136:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829997.791139:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787829997.791150:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829997/real 1787829997] req@ffff976889cfa940 x1874670564467840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829997.791151:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787829997.791171:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564467840/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829997.791172:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564468224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829997.791189:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829997.791194:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00080000:2.0:1787829997.791194:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829997.791198:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564467840:192.168.203.139@tcp:250:kworker.0 00000100:00080000:2.0:1787829997.791201:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829997.791204:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829997.791209:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4937 00000100:00080000:2.0:1787829997.791213:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829997.791217:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564467968:192.168.203.139@tcp:38:kworker.0 00000100:00080000:2.0:1787829997.791219:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787829997.791226:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829997.791236:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787829997.791242:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564468352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829997.791247:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829997/real 1787829997] req@ffff976889cfb480 x1874670564467968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829997.791265:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564467968/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829997.791275:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829997.791278:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829997.791280:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564467968:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829997.791284:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564468096:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787829997.791294:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829997.791299:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829997/real 1787829997] req@ffff976889cfad00 x1874670564468096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829997.791305:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564468096/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829997.791310:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829997.791311:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829997.791313:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564468096:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787829997.791316:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564468224:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829997.791324:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829997.791329:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829997/real 1787829997] req@ffff976889cfa1c0 x1874670564468224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829997.791335:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564468224/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829997.791339:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829997.791341:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829997.791343:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564468224:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787829997.791369:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564468352:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787829997.791388:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787829997.791399:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829997/real 1787829997] req@ffff976889cf8f00 x1874670564468352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829997.791409:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564468352/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829997.791414:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829997.791416:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787829997.791417:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564468352:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787829998.047009:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829998.048702:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829998.048713:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787829998.814902:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787829999.070916:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787829999.070937:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829999.071032:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787829999.838941:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787830000.094930:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787830000.094930:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787830000.095000:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830000.862904:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787830001.118935:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787830001.118998:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830001.119044:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787830001.887379:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787830002.142939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787830002.142951:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787830002.143010:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830002.910959:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830002.910973:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830002.910979:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787830002.910981:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787830002.910992:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4942 00000100:00080000:2.0:1787830002.911006:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830002.911016:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787830002.911130:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf92c0 x1874670564468480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830002.911159:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830002.911163:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830002.911165:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830002.911169:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4942 00000100:00080000:2.0:1787830002.911173:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830002.911178:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787830002.911191:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830002.911201:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564468480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787830002.911208:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf8f00 x1874670564468608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830002.911219:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830002.911222:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830002.911224:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830002.911228:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4942 00000100:00080000:2.0:1787830002.911231:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830002.911235:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787830002.911240:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830002.911250:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa1c0 x1874670564468736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830002.911260:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830002.911263:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830002.911266:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787830002.911271:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:2.0:1787830002.911275:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4942 00000100:00080000:2.0:1787830002.911279:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830002.911283:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787830002.911289:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787830002.911301:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfad00 x1874670564468864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830002.911310:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830002.911313:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787830002.911328:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830002/real 1787830002] req@ffff976889cf92c0 x1874670564468480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830002.911329:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830002.911337:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4942 00000100:00080000:2.0:1787830002.911343:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830002.911349:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787830002.911354:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830002.911358:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf92c0 x1874670564468480/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830002.911372:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830002.911377:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787830002.911381:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf92c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564468480:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787830002.911409:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564468608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787830002.911425:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb480 x1874670564468992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787830002.911453:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787830002.911469:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830002/real 1787830002] req@ffff976889cf8f00 x1874670564468608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830002.911483:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf8f00 x1874670564468608/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830002.911492:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830002.911496:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787830002.911499:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564468608:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787830002.911505:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564468736:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787830002.911524:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787830002.911535:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830002/real 1787830002] req@ffff976889cfa1c0 x1874670564468736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830002.911547:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa1c0 x1874670564468736/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830002.911555:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830002.911558:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787830002.911562:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564468736:192.168.203.139@tcp:38:kworker.0 00000100:00100000:0.0:1787830002.911568:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564468864:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787830002.911601:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787830002.911613:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830002/real 1787830002] req@ffff976889cfad00 x1874670564468864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830002.911623:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfad00 x1874670564468864/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830002.911631:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830002.911633:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787830002.911637:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564468864:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787830002.911652:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564468992:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787830002.911670:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787830002.911681:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830002/real 1787830002] req@ffff976889cfb480 x1874670564468992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830002.911692:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb480 x1874670564468992/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830002.911700:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830002.911703:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787830002.911715:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564468992:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787830003.166941:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787830003.167249:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830003.168509:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787830003.934939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787830004.190946:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787830004.191018:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787830004.191044:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830004.959088:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787830005.214913:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787830005.215657:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830005.216366:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787830005.983056:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787830006.238943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787830006.239128:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830006.239167:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787830007.007308:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787830007.262967:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787830007.263068:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787830007.263858:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830008.030972:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787830008.031008:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830008.031013:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830008.031017:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830008.031024:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4947 00000100:00080000:1.0:1787830008.031032:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830008.031038:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787830008.031370:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564469120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830008.031402:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830008.031407:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830008.031409:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830008.031414:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4947 00000100:00080000:1.0:1787830008.031418:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830008.031423:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787830008.031432:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830008.031446:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564469248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830008.031456:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830008.031459:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830008.031461:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830008.031465:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4947 00000100:00080000:1.0:1787830008.031468:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830008.031472:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787830008.031477:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830008.031487:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564469376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830008.031495:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830008.031498:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830008.031500:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830008.031504:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4947 00000100:00080000:1.0:1787830008.031507:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830008.031510:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787830008.031515:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787830008.031524:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564469504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830008.031533:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830008.031536:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830008.031538:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830008.031541:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4947 00000100:00080000:1.0:1787830008.031544:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830008.031547:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787830008.031551:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830008.031559:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564469632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830008.031637:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564469120:192.168.203.139@tcp:250:kworker.0 00000800:00000100:2.0:1787830008.031793:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787830008.031847:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830008/real 1787830008] req@ffff976889efc780 x1874670564469120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830008.031870:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564469120/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830008.031879:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830008.031883:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787830008.031887:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564469120:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787830008.031894:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564469248:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787830008.031917:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787830008.031929:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830008/real 1787830008] req@ffff976889eff480 x1874670564469248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830008.031941:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564469248/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830008.031949:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830008.031952:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787830008.031956:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564469248:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787830008.031961:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564469376:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787830008.031979:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787830008.031990:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830008/real 1787830008] req@ffff976889efe580 x1874670564469376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830008.032002:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564469376/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830008.032015:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830008.032019:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787830008.032022:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564469376:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787830008.032027:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564469504:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787830008.032045:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787830008.032055:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830008/real 1787830008] req@ffff976889efe1c0 x1874670564469504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830008.032067:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564469504/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830008.032075:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830008.032078:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787830008.032081:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564469504:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787830008.032086:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564469632:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787830008.032115:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787830008.032125:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830008/real 1787830008] req@ffff976889efcb40 x1874670564469632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830008.032201:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564469632/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830008.032210:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830008.032213:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787830008.032217:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564469632:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787830008.286939:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787830008.287414:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787830008.287455:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830009.054938:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787830009.310929:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787830009.311337:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830009.312310:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787830010.078939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787830010.334899:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787830010.334951:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787830010.335315:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830011.103008:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787830011.358928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787830011.358929:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787830011.359131:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830012.127016:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787830012.382936:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787830012.383039:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830012.383112:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00080000:1.0:1787830013.150928:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830013.150932:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787830013.150938:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830013.150943:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830013.150951:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4953 00000100:00080000:1.0:1787830013.150960:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830013.150968:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787830013.151002:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564469760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830013.151036:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830013.151039:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830013.151041:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830013.151045:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4953 00000100:00080000:1.0:1787830013.151047:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830013.151051:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787830013.151058:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830013.151068:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564469888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830013.151077:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830013.151080:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830013.151081:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830013.151084:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4953 00000100:00080000:1.0:1787830013.151087:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830013.151090:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787830013.151093:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830013.151100:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564470016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830013.151108:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830013.151111:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830013.151112:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830013.151116:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4953 00000100:00080000:1.0:1787830013.151119:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830013.151122:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787830013.151126:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787830013.151134:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564470144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830013.151141:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830013.151144:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830013.151146:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830013.151149:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4953 00000100:00080000:1.0:1787830013.151151:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830013.151154:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787830013.151157:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830013.151164:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564470272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830013.151285:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564469760:192.168.203.139@tcp:250:kworker.0 00000800:00000100:2.0:1787830013.151343:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787830013.151378:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830013/real 1787830013] req@ffff976889efe940 x1874670564469760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830013.151394:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564469760/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830013.151405:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830013.151408:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787830013.151411:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564469760:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787830013.151418:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564469888:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787830013.151437:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787830013.151448:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830013/real 1787830013] req@ffff976889eff0c0 x1874670564469888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830013.151460:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564469888/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830013.151469:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830013.151472:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787830013.151475:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564469888:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787830013.151481:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564470016:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787830013.151497:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787830013.151506:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830013/real 1787830013] req@ffff976889efc3c0 x1874670564470016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830013.151516:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564470016/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830013.151524:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830013.151527:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787830013.151530:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564470016:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787830013.151536:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564470144:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787830013.151553:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787830013.151563:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830013/real 1787830013] req@ffff976889effc00 x1874670564470144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830013.151573:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564470144/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830013.151580:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830013.151583:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787830013.151587:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564470144:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787830013.151592:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564470272:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787830013.151611:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787830013.151621:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830013/real 1787830013] req@ffff976889efed00 x1874670564470272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830013.151633:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564470272/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830013.151640:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830013.151643:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787830013.151646:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564470272:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787830013.406933:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787830013.406933:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787830013.408293:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830014.174919:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787830014.430955:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787830014.430977:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787830014.431077:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830015.199535:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787830015.454943:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787830015.455309:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787830015.456404:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830016.223422:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787830016.478984:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787830016.479409:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787830016.480724:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830017.246942:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787830017.502957:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787830017.503075:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787830017.503240:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830018.270910:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:0.0:1787830018.270992:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830018.270997:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830018.271001:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830018.271010:0:42426:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4958 00000100:00080000:0.0:1787830018.271020:0:42426:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830018.271027:0:42426:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:0.0:1787830018.271064:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee940 x1874670564470400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830018.271093:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830018.271097:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830018.271099:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830018.271102:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4958 00000100:00080000:0.0:1787830018.271104:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830018.271108:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787830018.271115:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830018.271128:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aef0c0 x1874670564470528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830018.271141:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830018.271145:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830018.271146:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830018.271150:0:42426:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4958 00000100:00080000:0.0:1787830018.271153:0:42426:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830018.271156:0:42426:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787830018.271161:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830018.271170:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aec3c0 x1874670564470656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830018.271177:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830018.271180:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830018.271181:0:42426:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830018.271184:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4958 00000100:00080000:0.0:1787830018.271187:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830018.271199:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:0.0:1787830018.271203:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1787830018.271225:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aee580 x1874670564470784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830018.271232:0:42426:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830018.271236:0:42426:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830018.271238:0:42426:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830018.271241:0:42426:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4958 00000100:00080000:0.0:1787830018.271244:0:42426:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830018.271248:0:42426:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787830018.271252:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564470400:192.168.203.139@tcp:250:kworker.0 00000008:00100000:0.0:1787830018.271255:0:42426:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830018.271267:0:42426:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976882aed680 x1874670564470912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787830018.271308:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787830018.271346:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830018/real 1787830018] req@ffff976882aee940 x1874670564470400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830018.271364:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee940 x1874670564470400/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830018.271372:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830018.271375:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787830018.271379:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564470400:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787830018.271385:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564470528:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787830018.271417:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787830018.271431:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830018/real 1787830018] req@ffff976882aef0c0 x1874670564470528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830018.271443:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aef0c0 x1874670564470528/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830018.271450:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830018.271453:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787830018.271455:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aef0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564470528:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787830018.271461:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564470656:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787830018.271480:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787830018.271491:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830018/real 1787830018] req@ffff976882aec3c0 x1874670564470656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830018.271501:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aec3c0 x1874670564470656/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830018.271514:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830018.271517:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787830018.271521:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aec3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564470656:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787830018.271526:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564470784:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787830018.271550:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787830018.271561:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830018/real 1787830018] req@ffff976882aee580 x1874670564470784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830018.271572:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aee580 x1874670564470784/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830018.271580:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830018.271583:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787830018.271586:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aee580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564470784:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787830018.271611:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564470912:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787830018.271631:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787830018.271641:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830018/real 1787830018] req@ffff976882aed680 x1874670564470912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830018.271651:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976882aed680 x1874670564470912/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830018.271659:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830018.271661:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787830018.271665:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976882aed680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564470912:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787830018.526937:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787830018.527396:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830018.527886:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787830019.295152:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787830019.550970:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787830019.551113:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787830019.551127:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830020.318933:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787830020.574940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787830020.575757:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830020.575995:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:2.0:1787830021.343532:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00010000:00010000:0.0:1787830021.599071:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830021.599146:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787830021.599203:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:2.0:1787830022.367881:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787830022.622937:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787830022.623064:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830022.623222:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:0.0:1787830023.391058:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:2.0:1787830023.391089:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830023.391095:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830023.391099:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830023.391108:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4963 00000100:00080000:2.0:1787830023.391118:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830023.391124:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787830023.391156:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa940 x1874670564471040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830023.391190:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830023.391194:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830023.391196:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830023.391200:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4963 00000100:00080000:2.0:1787830023.391204:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830023.391207:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787830023.391216:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830023.391227:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfbc00 x1874670564471168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830023.391237:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830023.391240:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830023.391241:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830023.391245:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4963 00000100:00100000:0.0:1787830023.391437:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564471040:192.168.203.139@tcp:250:kworker.0 00000800:00000100:0.0:1787830023.391495:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787830023.391532:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830023/real 1787830023] req@ffff976889cfa940 x1874670564471040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830023.391549:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa940 x1874670564471040/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830023.391560:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830023.391564:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787830023.391568:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564471040:192.168.203.139@tcp:250:kworker.0 00000100:00100000:0.0:1787830023.391584:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564471168:192.168.203.139@tcp:38:kworker.0 00000800:00000100:0.0:1787830023.391601:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787830023.391610:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830023/real 1787830023] req@ffff976889cfbc00 x1874670564471168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830023.391620:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfbc00 x1874670564471168/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830023.391628:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830023.391631:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787830023.391635:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfbc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564471168:192.168.203.139@tcp:38:kworker.0 00000100:00080000:2.0:1787830023.391904:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830023.391911:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787830023.391918:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830023.391931:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cf9e00 x1874670564471296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830023.392002:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830023.392006:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830023.392009:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830023.392013:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4963 00000100:00080000:2.0:1787830023.392018:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787830023.392023:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564471296:192.168.203.139@tcp:38:kworker.0 00000100:00080000:2.0:1787830023.392024:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787830023.392034:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1787830023.392048:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfa580 x1874670564471424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787830023.392051:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787830023.392061:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830023/real 1787830023] req@ffff976889cf9e00 x1874670564471296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830023.392083:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787830023.392085:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cf9e00 x1874670564471296/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830023.392088:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830023.392091:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830023.392095:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830023.392099:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4963 00000100:00080000:0.0:1787830023.392102:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787830023.392105:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cf9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564471296:192.168.203.139@tcp:38:kworker.0 00000100:00080000:2.0:1787830023.392105:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830023.392112:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:2.0:1787830023.392118:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830023.392120:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564471424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787830023.392133:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889cfb0c0 x1874670564471552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787830023.392139:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787830023.392149:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830023/real 1787830023] req@ffff976889cfa580 x1874670564471424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830023.392157:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfa580 x1874670564471424/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830023.392163:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830023.392166:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787830023.392168:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564471424:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787830023.392190:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564471552:192.168.203.139@tcp:8:kworker.0 00000800:00000100:0.0:1787830023.392208:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:0.0:1787830023.392219:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830023/real 1787830023] req@ffff976889cfb0c0 x1874670564471552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830023.392230:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889cfb0c0 x1874670564471552/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830023.392238:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830023.392240:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:0.0:1787830023.392243:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889cfb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564471552:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787830023.646967:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787830023.647233:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787830023.647372:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830024.415206:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787830024.670944:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787830024.670944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787830024.673103:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830025.439150:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787830025.694919:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:0.0:1787830025.695304:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830025.695370:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787830026.462946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787830026.718961:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787830026.718961:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787830026.719186:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830027.486946:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787830027.742938:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787830027.742945:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787830027.743044:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830028.510985:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830028.511170:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830028.511175:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787830028.511179:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787830028.511185:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4968 00000100:00080000:1.0:1787830028.511196:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830028.511203:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787830028.511234:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efed00 x1874670564471680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830028.511265:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830028.511276:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830028.511278:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830028.511281:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4968 00000100:00080000:1.0:1787830028.511285:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830028.511289:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787830028.511297:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830028.511309:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889effc00 x1874670564471808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830028.511321:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830028.511325:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830028.511326:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830028.511331:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4968 00000100:00080000:1.0:1787830028.511335:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787830028.511339:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564471680:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787830028.511342:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787830028.511349:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830028.511377:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc3c0 x1874670564471936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830028.511389:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830028.511393:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830028.511395:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830028.511400:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4968 00000100:00080000:1.0:1787830028.511405:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787830028.511405:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00080000:1.0:1787830028.511413:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787830028.511420:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787830028.511434:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff0c0 x1874670564472064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830028.511440:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830028/real 1787830028] req@ffff976889efed00 x1874670564471680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830028.511458:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830028.511462:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830028.511464:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830028.511471:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4968 00000100:00100000:2.0:1787830028.511471:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efed00 x1874670564471680/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830028.511476:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830028.511486:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00080000:2.0:1787830028.511489:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787830028.511492:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787830028.511496:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787830028.511501:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564471680:192.168.203.139@tcp:250:kworker.0 00000100:00100000:1.0:1787830028.511512:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe940 x1874670564472192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830028.511512:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564471808:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787830028.511543:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787830028.511557:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830028/real 1787830028] req@ffff976889effc00 x1874670564471808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830028.511570:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889effc00 x1874670564471808/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830028.511578:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830028.511582:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787830028.511585:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889effc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564471808:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787830028.511609:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564471936:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787830028.511630:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787830028.511640:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830028/real 1787830028] req@ffff976889efc3c0 x1874670564471936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830028.511651:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc3c0 x1874670564471936/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830028.511659:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830028.511662:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787830028.511666:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564471936:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787830028.511671:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564472064:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787830028.511690:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787830028.511700:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830028/real 1787830028] req@ffff976889eff0c0 x1874670564472064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830028.511710:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff0c0 x1874670564472064/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830028.511716:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830028.511718:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787830028.511720:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564472064:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787830028.511726:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564472192:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787830028.511758:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787830028.511768:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830028/real 1787830028] req@ffff976889efe940 x1874670564472192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830028.511779:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe940 x1874670564472192/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830028.511786:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830028.511789:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787830028.511793:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564472192:192.168.203.139@tcp:8:kworker.0 00000100:00100000:0.0:1787830028.766985:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787830028.767033:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787830028.767068:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830029.535387:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787830029.790959:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787830029.790965:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787830029.791146:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830030.558937:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787830030.814944:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787830030.815181:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787830030.815315:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830031.582978:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:1.0:1787830031.838940:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787830031.838995:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830031.839057:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787830032.606939:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787830032.862988:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787830032.863039:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787830032.863178:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830033.630951:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00080000:1.0:1787830033.631264:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 307626ff-a376-4faf-93e0-9f55b723d964->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830033.631271:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830033.631275:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830033.631284:0:56784:0:(import.c:510:import_select_connection()) MGC192.168.203.139@tcp: connect to NID 192.168.203.139@tcp last attempt 4973 00000100:00080000:1.0:1787830033.631295:0:56784:0:(import.c:589:import_select_connection()) MGC192.168.203.139@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830033.631302:0:56784:0:(import.c:624:import_select_connection()) MGC192.168.203.139@tcp: import ffff97688929d800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:1.0:1787830033.631339:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efcb40 x1874670564472320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830033.631369:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830033.631372:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830033.631374:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830033.631378:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4973 00000100:00080000:1.0:1787830033.631382:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830033.631385:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9768877a4800: import ffff976889e8e800 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787830033.631394:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830033.631408:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe1c0 x1874670564472448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830033.631418:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830033.631422:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787830033.631422:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564472320:192.168.203.139@tcp:250:kworker.0 00000100:00080000:1.0:1787830033.631425:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830033.631432:0:56784:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4973 00000100:00080000:1.0:1787830033.631436:0:56784:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830033.631439:0:56784:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9768877a4800: import ffff97688929c000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000008:00100000:1.0:1787830033.631444:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830033.631454:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efe580 x1874670564472576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830033.631462:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830033.631465:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830033.631467:0:56784:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830033.631470:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4973 00000100:00080000:1.0:1787830033.631481:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830033.631486:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9768877a4800: import ffff976889298000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000800:00000100:2.0:1787830033.631491:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000008:00100000:1.0:1787830033.631494:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1787830033.631507:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889eff480 x1874670564472704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830033.631516:0:56784:0:(pinger.c:229:ptlrpc_pinger_process_import()) 09ed01c7-7b8c-4e06-9d82-a6b70e42014a->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830033.631520:0:56784:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830033.631522:0:56784:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830033.631525:0:56784:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: connect to NID 192.168.203.139@tcp last attempt 4973 00000100:00080000:1.0:1787830033.631529:0:56784:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830033.631535:0:56784:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9768877a4800: import ffff97688929b000 using connection 192.168.203.139@tcp/192.168.203.139@tcp 00000100:00100000:2.0:1787830033.631536:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830033/real 1787830033] req@ffff976889efcb40 x1874670564472320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787830033.631554:0:56784:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830033.631554:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efcb40 x1874670564472320/t0(0) o250->MGC192.168.203.139@tcp@192.168.203.139@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830033.631569:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787830033.631573:0:56784:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff976889efc780 x1874670564472832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830033.631576:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787830033.631585:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:307626ff-a376-4faf-93e0-9f55b723d964:2400:1874670564472320:192.168.203.139@tcp:250:kworker.0 00000100:00100000:2.0:1787830033.631614:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564472448:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787830033.631640:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787830033.631664:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830033/real 1787830033] req@ffff976889efe1c0 x1874670564472448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830033.631675:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe1c0 x1874670564472448/t0(0) o38->lustre-MDT0000-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830033.631683:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889e8e800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830033.631685:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787830033.631688:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564472448:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787830033.631694:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564472576:192.168.203.139@tcp:38:kworker.0 00000800:00000100:2.0:1787830033.631711:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787830033.631720:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830033/real 1787830033] req@ffff976889efe580 x1874670564472576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830033.631738:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efe580 x1874670564472576/t0(0) o38->lustre-MDT0001-mdc-ffff9768877a4800@192.168.203.139@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830033.631746:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929c000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830033.631749:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787830033.631752:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efe580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564472576:192.168.203.139@tcp:38:kworker.0 00000100:00100000:2.0:1787830033.631757:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564472704:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787830033.631775:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787830033.631785:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830033/real 1787830033] req@ffff976889eff480 x1874670564472704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830033.631795:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889eff480 x1874670564472704/t0(0) o8->lustre-OST0000-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830033.631802:0:2400:0:(import.c:65:import_set_state_nolock()) ffff976889298000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830033.631805:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787830033.631808:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889eff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564472704:192.168.203.139@tcp:8:kworker.0 00000100:00100000:2.0:1787830033.631812:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564472832:192.168.203.139@tcp:8:kworker.0 00000800:00000100:2.0:1787830033.631850:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.139@tcp 00000100:00100000:2.0:1787830033.631860:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830033/real 1787830033] req@ffff976889efc780 x1874670564472832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830033.631870:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff976889efc780 x1874670564472832/t0(0) o8->lustre-OST0001-osc-ffff9768877a4800@192.168.203.139@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830033.631876:0:2400:0:(import.c:65:import_set_state_nolock()) ffff97688929b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830033.631879:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.139@tcp failed (-110) 00000100:00100000:2.0:1787830033.631882:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff976889efc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:09ed01c7-7b8c-4e06-9d82-a6b70e42014a:2400:1874670564472832:192.168.203.139@tcp:8:kworker.0 00000100:00100000:1.0:1787830033.886978:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:0.0:1787830033.887096:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830033.887134:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:3.0:1787830034.654944:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:2.0:1787830034.910928:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:1.0:1787830034.911172:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830034.911221:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:3.0:1787830035.678943:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787830035.934926:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00000100:00100000:0.0:1787830035.935148:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00010000:00010000:2.0:1787830035.935167:0:56503:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830036.703099:0:56419:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6240 going to sleep for 0 seconds 00000100:00100000:0.0:1787830036.959064:0:42220:0:(client.c:2715:ptlrpc_set_wait()) set ffff9768984f6180 going to sleep for 0 seconds 00000100:00100000:1.0:1787830036.959092:0:49796:0:(client.c:2715:ptlrpc_set_wait()) set ffff97688cb3e300 going to sleep for 0 seconds 00010000:00010000:2.0:1787830036.959128:0:56500:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 77816 lines, 77816 kept, 0 dropped, 0 bad.